Nov 03 17:42:33.250543 ip-10-0-128-204 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Nov 03 17:42:33.250555 ip-10-0-128-204 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Nov 03 17:42:33.250562 ip-10-0-128-204 systemd[1]: kubelet.service: Failed with result 'resources'.
Nov 03 17:42:33.250824 ip-10-0-128-204 systemd[1]: Failed to start Kubernetes Kubelet.
Nov 03 17:42:43.304186 ip-10-0-128-204 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Nov 03 17:42:43.304202 ip-10-0-128-204 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 1fe403c307d445e781940505140c7d13 --
Nov 03 17:44:48.903949 ip-10-0-128-204 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 17:44:49.321700 ip-10-0-128-204 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 17:44:49.321700 ip-10-0-128-204 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 17:44:49.321700 ip-10-0-128-204 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 17:44:49.321700 ip-10-0-128-204 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 17:44:49.326319 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326227    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:44:49.326319 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326314    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:44:49.326319 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326318    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:44:49.326319 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326321    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:44:49.326319 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326324    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:44:49.326319 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326327    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:44:49.326536 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326330    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:44:49.326536 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326333    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:44:49.326536 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326336    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:44:49.326536 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326338    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:44:49.326536 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326342    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:44:49.326536 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326345    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:44:49.326536 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326347    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:44:49.326536 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326350    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:44:49.326536 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326353    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:44:49.326536 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326355    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:44:49.326536 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326358    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:44:49.326536 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326361    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:44:49.326536 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326364    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:44:49.326536 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326366    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:44:49.326536 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326369    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:44:49.326536 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326379    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:44:49.326536 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326382    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:44:49.326536 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326385    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:44:49.326536 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326387    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:44:49.327034 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326390    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:44:49.327034 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326393    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:44:49.327034 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326395    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:44:49.327034 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326398    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:44:49.327034 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326401    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:44:49.327034 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326403    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:44:49.327034 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326406    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:44:49.327034 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326408    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:44:49.327034 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326411    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:44:49.327034 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326413    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:44:49.327034 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326416    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:44:49.327034 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326419    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:44:49.327034 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326421    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:44:49.327034 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326424    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:44:49.327034 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326428    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:44:49.327034 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326431    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:44:49.327034 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326434    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:44:49.327034 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326437    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:44:49.327034 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326440    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:44:49.327034 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326443    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:44:49.327504 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326446    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:44:49.327504 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326448    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:44:49.327504 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326451    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:44:49.327504 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326454    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:44:49.327504 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326456    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:44:49.327504 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326460    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:44:49.327504 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326463    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:44:49.327504 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326466    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:44:49.327504 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326468    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:44:49.327504 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326471    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:44:49.327504 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326474    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:44:49.327504 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326476    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:44:49.327504 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326479    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:44:49.327504 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326482    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:44:49.327504 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326485    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:44:49.327504 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326487    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:44:49.327504 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326490    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:44:49.327504 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326492    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:44:49.327504 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326495    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:44:49.327968 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326498    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:44:49.327968 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326500    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:44:49.327968 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326506    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:44:49.327968 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326511    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:44:49.327968 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326514    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:44:49.327968 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326517    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:44:49.327968 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326521    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:44:49.327968 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326524    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:44:49.327968 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326527    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:44:49.327968 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326529    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:44:49.327968 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326532    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:44:49.327968 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326535    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:44:49.327968 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326538    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:44:49.327968 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326541    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:44:49.327968 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326544    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:44:49.327968 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326547    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:44:49.327968 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326549    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:44:49.327968 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326552    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:44:49.327968 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326555    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:44:49.328417 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326560    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:44:49.328417 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326567    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:44:49.328417 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326572    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:44:49.328417 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326575    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:44:49.328417 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326578    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:44:49.328417 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326581    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:44:49.328417 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326584    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:44:49.328417 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326587    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:44:49.328417 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326590    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:44:49.328417 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326593    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:44:49.328417 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326595    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:44:49.328417 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326598    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:44:49.328417 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326601    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:44:49.328417 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326603    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:44:49.328417 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326619    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:44:49.328417 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326624    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:44:49.328417 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326632    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:44:49.328417 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326635    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:44:49.328417 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326638    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:44:49.328886 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326641    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:44:49.328886 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326644    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:44:49.328886 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326647    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:44:49.328886 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.326649    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:44:49.328886 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327448    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:44:49.328886 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327456    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:44:49.328886 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327459    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:44:49.328886 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327462    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:44:49.328886 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327466    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:44:49.328886 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327469    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:44:49.328886 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327472    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:44:49.328886 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327475    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:44:49.328886 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327478    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:44:49.328886 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327481    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:44:49.328886 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327490    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:44:49.328886 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327493    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:44:49.328886 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327497    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:44:49.328886 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327501    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:44:49.328886 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327504    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:44:49.328886 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327507    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:44:49.329392 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327510    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:44:49.329392 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327513    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:44:49.329392 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327515    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:44:49.329392 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327518    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:44:49.329392 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327521    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:44:49.329392 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327523    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:44:49.329392 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327526    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:44:49.329392 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327528    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:44:49.329392 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327531    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:44:49.329392 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327534    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:44:49.329392 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327537    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:44:49.329392 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327539    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:44:49.329392 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327542    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:44:49.329392 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327547    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:44:49.329392 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327551    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:44:49.329392 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327554    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:44:49.329392 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327556    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:44:49.329392 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327559    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:44:49.329392 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327562    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:44:49.329860 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327566    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:44:49.329860 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327569    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:44:49.329860 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327571    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:44:49.329860 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327574    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:44:49.329860 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327576    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:44:49.329860 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327579    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:44:49.329860 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327582    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:44:49.329860 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327584    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:44:49.329860 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327588    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:44:49.329860 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327591    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:44:49.329860 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327594    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:44:49.329860 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327596    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:44:49.329860 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327599    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:44:49.329860 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327602    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:44:49.329860 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327604    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:44:49.329860 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327626    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:44:49.329860 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327631    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:44:49.329860 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327634    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:44:49.329860 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327636    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:44:49.329860 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327639    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:44:49.330335 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327642    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:44:49.330335 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327645    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:44:49.330335 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327648    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:44:49.330335 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327650    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:44:49.330335 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327654    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:44:49.330335 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327656    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:44:49.330335 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327659    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:44:49.330335 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327662    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:44:49.330335 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327665    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:44:49.330335 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327668    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:44:49.330335 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327670    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:44:49.330335 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327673    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:44:49.330335 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327676    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:44:49.330335 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327678    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:44:49.330335 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327681    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:44:49.330335 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327684    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:44:49.330335 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327686    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:44:49.330335 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327689    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:44:49.330335 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327692    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:44:49.330803 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327694    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:44:49.330803 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327697    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:44:49.330803 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327700    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:44:49.330803 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327703    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:44:49.330803 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327705    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:44:49.330803 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327708    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:44:49.330803 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327710    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:44:49.330803 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327713    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:44:49.330803 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327716    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:44:49.330803 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327718    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:44:49.330803 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327721    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:44:49.330803 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327723    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:44:49.330803 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327726    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:44:49.330803 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327729    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:44:49.330803 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327732    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:44:49.330803 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327734    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:44:49.330803 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327737    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:44:49.330803 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327740    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:44:49.330803 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327743    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:44:49.331254 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327746    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:44:49.331254 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327748    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:44:49.331254 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327751    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:44:49.331254 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327753    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:44:49.331254 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327756    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:44:49.331254 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327758    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:44:49.331254 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327761    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:44:49.331254 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327764    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:44:49.331254 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327766    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:44:49.331254 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327769    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:44:49.331254 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327772    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:44:49.331254 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327775    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:44:49.331254 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.327779    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:44:49.331254 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328296    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:44:49.331254 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328302    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:44:49.331254 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328306    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:44:49.331254 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328310    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:44:49.331254 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328313    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:44:49.331254 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328316    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:44:49.331254 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328318    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:44:49.331747 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328321    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:44:49.331747 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328324    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:44:49.331747 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328327    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:44:49.331747 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328329    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:44:49.331747 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328332    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:44:49.331747 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328335    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:44:49.331747 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328337    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:44:49.331747 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328341    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:44:49.331747 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328344    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:44:49.331747 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328346    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:44:49.331747 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328349    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:44:49.331747 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328352    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:44:49.331747 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328355    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:44:49.331747 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328358    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:44:49.331747 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328361    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:44:49.331747 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328363    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:44:49.331747 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328366    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:44:49.331747 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328369    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:44:49.331747 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328371    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:44:49.331747 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328374    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:44:49.332206 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328377    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:44:49.332206 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328379    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:44:49.332206 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328382    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:44:49.332206 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328384    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:44:49.332206 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328388    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:44:49.332206 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328392    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:44:49.332206 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328395    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:44:49.332206 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328398    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:44:49.332206 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328407    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:44:49.332206 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328410    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:44:49.332206 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328413    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:44:49.332206 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328416    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:44:49.332206 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328418    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:44:49.332206 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328421    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:44:49.332206 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328424    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:44:49.332206 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328427    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:44:49.332206 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328430    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:44:49.332206 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328433    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:44:49.332206 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328435    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:44:49.332667 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328438    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:44:49.332667 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328441    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:44:49.332667 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328443    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:44:49.332667 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328446    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:44:49.332667 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328449    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:44:49.332667 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328451    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:44:49.332667 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328454    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:44:49.332667 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328456    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:44:49.332667 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328459    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:44:49.332667 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328462    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:44:49.332667 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328464    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:44:49.332667 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328467    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:44:49.332667 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328470    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:44:49.332667 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328473    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:44:49.332667 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328475    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:44:49.332667 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328478    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:44:49.332667 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328480    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:44:49.332667 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328483    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:44:49.332667 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328485    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:44:49.332667 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328488    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:44:49.333142 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328491    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:44:49.333142 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328493    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:44:49.333142 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328502    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:44:49.333142 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328504    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:44:49.333142 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328507    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:44:49.333142 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328510    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:44:49.333142 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328512    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:44:49.333142 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328515    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:44:49.333142 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328520    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:44:49.333142 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328523    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:44:49.333142 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328526    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:44:49.333142 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328528    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:44:49.333142 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328531    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:44:49.333142 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328534    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:44:49.333142 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328537    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:44:49.333142 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328540    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:44:49.333142 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328544    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:44:49.333142 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328548    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:44:49.333142 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328551    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:44:49.333587 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328553    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:44:49.333587 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328556    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:44:49.333587 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328558    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:44:49.333587 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328561    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:44:49.333587 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328564    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:44:49.333587 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328566    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:44:49.333587 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328569    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:44:49.333587 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328572    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:44:49.333587 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328574    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:44:49.333587 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328577    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:44:49.333587 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328580    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:44:49.333587 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328583    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:44:49.333587 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328585    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:44:49.333587 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328588    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:44:49.333587 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328590    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:44:49.333587 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328593    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:44:49.333587 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328596    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:44:49.333587 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328599    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:44:49.333587 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328601    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:44:49.333587 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328605    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:44:49.334062 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.328622    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:44:49.334062 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329531    2576 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 17:44:49.334062 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329541    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 17:44:49.334062 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329549    2576 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 17:44:49.334062 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329556    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 17:44:49.334062 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329561    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 17:44:49.334062 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329565    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 17:44:49.334062 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329569    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 17:44:49.334062 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329573    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 17:44:49.334062 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329576    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 17:44:49.334062 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329580    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 17:44:49.334062 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329583    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 17:44:49.334062 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329587    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 17:44:49.334062 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329590    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 17:44:49.334062 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329593    2576 flags.go:64] FLAG: --cgroup-root=""
Nov 03 17:44:49.334062 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329596    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 17:44:49.334062 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329599    2576 flags.go:64] FLAG: --client-ca-file=""
Nov 03 17:44:49.334062 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329602    2576 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 17:44:49.334062 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329625    2576 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 17:44:49.334062 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329631    2576 flags.go:64] FLAG: --cluster-domain=""
Nov 03 17:44:49.334062 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329634    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 17:44:49.334062 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329637    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 17:44:49.334062 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329640    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 17:44:49.334062 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329644    2576 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 17:44:49.334677 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329652    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 17:44:49.334677 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329655    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 17:44:49.334677 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329658    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 17:44:49.334677 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329662    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 17:44:49.334677 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329665    2576 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 17:44:49.334677 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329668    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 17:44:49.334677 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329671    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 17:44:49.334677 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329674    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 17:44:49.334677 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329677    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 17:44:49.334677 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329682    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 17:44:49.334677 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329686    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 17:44:49.334677 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329688    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 17:44:49.334677 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329693    2576 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 17:44:49.334677 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329696    2576 flags.go:64] FLAG: --enable-server="true"
Nov 03 17:44:49.334677 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329699    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 17:44:49.334677 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329704    2576 flags.go:64] FLAG: --event-burst="100"
Nov 03 17:44:49.334677 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329707    2576 flags.go:64] FLAG: --event-qps="50"
Nov 03 17:44:49.334677 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329710    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 17:44:49.334677 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329713    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 17:44:49.334677 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329716    2576 flags.go:64] FLAG: --eviction-hard=""
Nov 03 17:44:49.334677 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329721    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 17:44:49.334677 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329724    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 17:44:49.334677 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329727    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 17:44:49.334677 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329730    2576 flags.go:64] FLAG: --eviction-soft=""
Nov 03 17:44:49.334677 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329733    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 17:44:49.335282 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329736    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 17:44:49.335282 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329739    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 17:44:49.335282 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329741    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 17:44:49.335282 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329744    2576 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 17:44:49.335282 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329747    2576 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 17:44:49.335282 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329751    2576 flags.go:64] FLAG: --feature-gates=""
Nov 03 17:44:49.335282 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329755    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 17:44:49.335282 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329758    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 17:44:49.335282 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329761    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 17:44:49.335282 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329764    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 17:44:49.335282 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329767    2576 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 17:44:49.335282 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329770    2576 flags.go:64] FLAG: --help="false"
Nov 03 17:44:49.335282 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329773    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-128-204.us-west-2.compute.internal"
Nov 03 17:44:49.335282 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329777    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 17:44:49.335282 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329780    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 17:44:49.335282 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329783    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 17:44:49.335282 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329787    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 17:44:49.335282 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329791    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 17:44:49.335282 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329794    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 17:44:49.335282 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329799    2576 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 17:44:49.335282 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329802    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 17:44:49.335282 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329805    2576 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 17:44:49.335282 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329808    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 17:44:49.335882 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329811    2576 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 17:44:49.335882 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329814    2576 flags.go:64] FLAG: --kube-reserved=""
Nov 03 17:44:49.335882 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329817    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 17:44:49.335882 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329820    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 17:44:49.335882 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329823    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 17:44:49.335882 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329826    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 17:44:49.335882 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329829    2576 flags.go:64] FLAG: --lock-file=""
Nov 03 17:44:49.335882 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329832    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 17:44:49.335882 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329835    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 17:44:49.335882 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329838    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 17:44:49.335882 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329843    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 17:44:49.335882 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329846    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 17:44:49.335882 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329849    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 17:44:49.335882 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329852    2576 flags.go:64] FLAG: --logging-format="text"
Nov 03 17:44:49.335882 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329855    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 17:44:49.335882 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329859    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 17:44:49.335882 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329862    2576 flags.go:64] FLAG: --manifest-url=""
Nov 03 17:44:49.335882 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329864    2576 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 17:44:49.335882 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329869    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 17:44:49.335882 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329872    2576 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 17:44:49.335882 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329876    2576 flags.go:64] FLAG: --max-pods="110"
Nov 03 17:44:49.335882 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329879    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 17:44:49.335882 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329882    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 17:44:49.335882 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329888    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 17:44:49.335882 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329891    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 17:44:49.336472 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329895    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 17:44:49.336472 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329898    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 17:44:49.336472 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329900    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 17:44:49.336472 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329909    2576 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 17:44:49.336472 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329913    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 17:44:49.336472 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329916    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 17:44:49.336472 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329919    2576 flags.go:64] FLAG: --pod-cidr=""
Nov 03 17:44:49.336472 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329922    2576 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 17:44:49.336472 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329925    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 17:44:49.336472 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329928    2576 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 17:44:49.336472 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329931    2576 flags.go:64] FLAG: --port="10250"
Nov 03 17:44:49.336472 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329933    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 17:44:49.336472 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329936    2576 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0e294fd8eca1f506e"
Nov 03 17:44:49.336472 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329939    2576 flags.go:64] FLAG: --qos-reserved=""
Nov 03 17:44:49.336472 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329943    2576 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 17:44:49.336472 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329945    2576 flags.go:64] FLAG: --register-node="true"
Nov 03 17:44:49.336472 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329948    2576 flags.go:64] FLAG: --register-with-taints=""
Nov 03 17:44:49.336472 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329952    2576 flags.go:64] FLAG: --registry-burst="10"
Nov 03 17:44:49.336472 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329955    2576 flags.go:64] FLAG: --registry-qps="5"
Nov 03 17:44:49.336472 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329958    2576 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 17:44:49.336472 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329961    2576 flags.go:64] FLAG: --reserved-memory=""
Nov 03 17:44:49.336472 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329965    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 17:44:49.336472 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329968    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 17:44:49.336472 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329971    2576 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 17:44:49.336472 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329974    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 17:44:49.336472 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329977    2576 flags.go:64] FLAG: --runonce="false"
Nov 03 17:44:49.337175 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329980    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 17:44:49.337175 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329983    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 17:44:49.337175 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329986    2576 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 17:44:49.337175 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329989    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 17:44:49.337175 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329992    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 17:44:49.337175 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329995    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 17:44:49.337175 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.329999    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 17:44:49.337175 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.330002    2576 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 17:44:49.337175 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.330004    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 17:44:49.337175 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.330009    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 17:44:49.337175 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.330012    2576 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 17:44:49.337175 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.330015    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 17:44:49.337175 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.330018    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 17:44:49.337175 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.330021    2576 flags.go:64] FLAG: --system-cgroups=""
Nov 03 17:44:49.337175 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.330024    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 17:44:49.337175 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.330030    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 17:44:49.337175 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.330033    2576 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 17:44:49.337175 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.330035    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 17:44:49.337175 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.330039    2576 flags.go:64] FLAG: --tls-min-version=""
Nov 03 17:44:49.337175 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.330042    2576 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 17:44:49.337175 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.330045    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 17:44:49.337175 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.330048    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 17:44:49.337175 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.330051    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 17:44:49.337175 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.330054    2576 flags.go:64] FLAG: --v="2"
Nov 03 17:44:49.337175 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.330058    2576 flags.go:64] FLAG: --version="false"
Nov 03 17:44:49.337776 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.330062    2576 flags.go:64] FLAG: --vmodule=""
Nov 03 17:44:49.337776 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.330066    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 17:44:49.337776 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.330069    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 17:44:49.337776 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330205    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:44:49.337776 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330209    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:44:49.337776 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330213    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:44:49.337776 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330216    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:44:49.337776 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330219    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:44:49.337776 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330224    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:44:49.337776 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330226    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:44:49.337776 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330229    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:44:49.337776 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330232    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:44:49.337776 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330234    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:44:49.337776 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330238    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:44:49.337776 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330241    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:44:49.337776 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330244    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:44:49.337776 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330248    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:44:49.337776 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330251    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:44:49.337776 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330253    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:44:49.337776 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330256    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:44:49.338290 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330259    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:44:49.338290 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330261    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:44:49.338290 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330264    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:44:49.338290 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330267    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:44:49.338290 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330269    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:44:49.338290 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330272    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:44:49.338290 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330275    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:44:49.338290 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330277    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:44:49.338290 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330280    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:44:49.338290 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330282    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:44:49.338290 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330285    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:44:49.338290 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330289    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:44:49.338290 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330293    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:44:49.338290 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330296    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:44:49.338290 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330299    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:44:49.338290 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330301    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:44:49.338290 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330304    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:44:49.338290 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330307    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:44:49.338290 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330310    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:44:49.338290 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330312    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:44:49.338788 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330316    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:44:49.338788 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330319    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:44:49.338788 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330322    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:44:49.338788 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330324    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:44:49.338788 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330327    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:44:49.338788 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330330    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:44:49.338788 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330333    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:44:49.338788 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330336    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:44:49.338788 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330340    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:44:49.338788 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330343    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:44:49.338788 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330346    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:44:49.338788 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330349    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:44:49.338788 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330351    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:44:49.338788 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330354    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:44:49.338788 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330357    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:44:49.338788 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330359    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:44:49.338788 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330362    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:44:49.338788 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330365    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:44:49.338788 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330367    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:44:49.338788 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330369    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:44:49.339257 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330372    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:44:49.339257 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330375    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:44:49.339257 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330377    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:44:49.339257 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330380    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:44:49.339257 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330383    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:44:49.339257 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330386    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:44:49.339257 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330389    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:44:49.339257 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330391    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:44:49.339257 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330394    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:44:49.339257 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330397    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:44:49.339257 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330399    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:44:49.339257 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330402    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:44:49.339257 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330406    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:44:49.339257 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330409    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:44:49.339257 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330411    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:44:49.339257 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330414    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:44:49.339257 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330416    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:44:49.339257 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330419    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:44:49.339257 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330422    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:44:49.339257 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330425    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:44:49.339809 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330428    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:44:49.339809 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330431    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:44:49.339809 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330434    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:44:49.339809 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330436    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:44:49.339809 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330439    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:44:49.339809 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330442    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:44:49.339809 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330445    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:44:49.339809 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330449    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:44:49.339809 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330452    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:44:49.339809 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330455    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:44:49.339809 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330457    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:44:49.339809 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330460    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:44:49.339809 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330463    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:44:49.339809 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330465    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:44:49.339809 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330468    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:44:49.339809 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330471    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:44:49.339809 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330473    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:44:49.339809 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330476    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:44:49.339809 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330479    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:44:49.340254 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330482    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:44:49.340254 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330484    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:44:49.340254 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330487    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:44:49.340254 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330489    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:44:49.340254 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330492    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:44:49.340254 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330495    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:44:49.340254 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330498    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:44:49.340254 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330501    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:44:49.340254 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330503    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:44:49.340254 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330506    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:44:49.340254 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.330532    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 17:44:49.340254 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330572    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:44:49.340254 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330576    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:44:49.340254 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330580    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:44:49.340254 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330582    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:44:49.340254 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330585    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:44:49.340653 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330588    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:44:49.340653 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330591    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:44:49.340653 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330593    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:44:49.340653 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330596    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:44:49.340653 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330599    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:44:49.340653 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330601    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:44:49.340653 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330604    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:44:49.340653 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330619    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:44:49.340653 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330622    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:44:49.340653 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330624    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:44:49.340653 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330628    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:44:49.340653 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330631    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:44:49.340653 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330634    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:44:49.340653 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330636    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:44:49.340653 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330639    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:44:49.340653 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330642    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:44:49.340653 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330645    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:44:49.340653 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330647    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:44:49.340653 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330650    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:44:49.341109 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330653    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:44:49.341109 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330655    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:44:49.341109 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330658    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:44:49.341109 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330662    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:44:49.341109 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330664    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:44:49.341109 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330667    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:44:49.341109 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330671    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:44:49.341109 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330675    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:44:49.341109 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330678    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:44:49.341109 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330681    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:44:49.341109 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330684    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:44:49.341109 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330688    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:44:49.341109 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330691    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:44:49.341109 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330693    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:44:49.341109 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330696    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:44:49.341109 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330698    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:44:49.341109 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330701    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:44:49.341109 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330704    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:44:49.341109 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330706    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:44:49.341109 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330709    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:44:49.341570 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330712    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:44:49.341570 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330714    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:44:49.341570 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330717    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:44:49.341570 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330720    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:44:49.341570 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330722    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:44:49.341570 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330725    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:44:49.341570 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330728    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:44:49.341570 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330730    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:44:49.341570 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330733    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:44:49.341570 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330736    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:44:49.341570 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330738    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:44:49.341570 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330741    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:44:49.341570 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330743    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:44:49.341570 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330746    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:44:49.341570 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330748    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:44:49.341570 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330752    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:44:49.341570 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330754    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:44:49.341570 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330757    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:44:49.341570 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330760    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:44:49.341570 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330763    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:44:49.342047 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330765    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:44:49.342047 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330768    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:44:49.342047 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330771    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:44:49.342047 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330774    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:44:49.342047 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330777    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:44:49.342047 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330780    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:44:49.342047 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330782    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:44:49.342047 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330785    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:44:49.342047 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330788    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:44:49.342047 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330790    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:44:49.342047 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330793    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:44:49.342047 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330795    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:44:49.342047 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330798    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:44:49.342047 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330801    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:44:49.342047 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330804    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:44:49.342047 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330806    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:44:49.342047 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330809    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:44:49.342047 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330812    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:44:49.342047 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330815    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:44:49.342490 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330818    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:44:49.342490 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330821    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:44:49.342490 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330825    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:44:49.342490 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330828    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:44:49.342490 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330831    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:44:49.342490 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330834    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:44:49.342490 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330837    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:44:49.342490 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330839    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:44:49.342490 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330843    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:44:49.342490 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330846    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:44:49.342490 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330849    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:44:49.342490 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330851    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:44:49.342490 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330854    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:44:49.342490 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330856    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:44:49.342490 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330860    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:44:49.342490 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330863    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:44:49.342490 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330865    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:44:49.342490 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330868    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:44:49.342490 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330872    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:44:49.342946 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330874    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:44:49.342946 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330877    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:44:49.342946 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330893    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:44:49.342946 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.330897    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:44:49.342946 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.337784    2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 17:44:49.342946 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.337800    2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 17:44:49.342946 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.337817    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 17:44:49.342946 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.337822    2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 17:44:49.342946 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.337888    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:44:49.342946 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.337894    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:44:49.342946 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.337897    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:44:49.342946 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.337902    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:44:49.342946 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.337905    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:44:49.342946 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.337908    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:44:49.342946 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.337911    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:44:49.342946 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.337914    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:44:49.342946 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.337917    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:44:49.342946 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.337920    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:44:49.342946 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.337922    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:44:49.343417 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.337925    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:44:49.343417 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.337927    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:44:49.343417 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.337932    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:44:49.343417 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.337935    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:44:49.343417 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.337938    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:44:49.343417 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.337941    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:44:49.343417 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.337944    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:44:49.343417 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.337947    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:44:49.343417 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.337949    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:44:49.343417 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.337952    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:44:49.343417 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.337955    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:44:49.343417 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.337957    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:44:49.343417 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.337961    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:44:49.343417 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.337964    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:44:49.343417 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.337967    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:44:49.343417 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.337969    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:44:49.343417 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.337973    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:44:49.343417 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.337975    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:44:49.343417 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.337978    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:44:49.343417 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.337981    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:44:49.343963 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.337983    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:44:49.343963 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.337986    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:44:49.343963 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.337989    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:44:49.343963 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.337992    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:44:49.343963 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.337994    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:44:49.343963 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.337997    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:44:49.343963 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.337999    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:44:49.343963 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338002    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:44:49.343963 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338005    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:44:49.343963 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338008    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:44:49.343963 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338010    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:44:49.343963 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338013    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:44:49.343963 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338015    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:44:49.343963 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338018    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:44:49.343963 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338021    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:44:49.343963 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338024    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:44:49.343963 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338026    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:44:49.343963 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338029    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:44:49.343963 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338032    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:44:49.343963 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338034    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:44:49.344429 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338037    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:44:49.344429 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338039    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:44:49.344429 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338042    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:44:49.344429 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338045    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:44:49.344429 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338047    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:44:49.344429 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338051    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:44:49.344429 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338054    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:44:49.344429 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338057    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:44:49.344429 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338059    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:44:49.344429 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338062    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:44:49.344429 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338065    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:44:49.344429 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338068    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:44:49.344429 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338072    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:44:49.344429 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338075    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:44:49.344429 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338079    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:44:49.344429 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338082    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:44:49.344429 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338085    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:44:49.344429 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338088    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:44:49.344429 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338092    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:44:49.344887 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338094    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:44:49.344887 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338097    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:44:49.344887 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338100    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:44:49.344887 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338103    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:44:49.344887 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338105    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:44:49.344887 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338108    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:44:49.344887 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338111    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:44:49.344887 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338113    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:44:49.344887 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338116    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:44:49.344887 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338119    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:44:49.344887 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338122    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:44:49.344887 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338125    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:44:49.344887 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338127    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:44:49.344887 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338130    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:44:49.344887 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338133    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:44:49.344887 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338135    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:44:49.344887 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338138    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:44:49.344887 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338141    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:44:49.344887 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338144    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:44:49.344887 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338151    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:44:49.345394 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338154    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:44:49.345394 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338157    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:44:49.345394 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338160    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:44:49.345394 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338163    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:44:49.345394 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338165    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:44:49.345394 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338168    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:44:49.345394 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338176    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:44:49.345394 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338179    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:44:49.345394 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338182    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:44:49.345394 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338184    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:44:49.345394 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338187    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:44:49.345394 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338189    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:44:49.345394 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338192    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:44:49.345394 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338195    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:44:49.345394 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338197    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:44:49.345394 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338200    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:44:49.345795 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.338227    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 17:44:49.345795 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338407    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:44:49.345795 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338412    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:44:49.345795 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338415    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:44:49.345795 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338419    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:44:49.345795 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338422    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:44:49.345795 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338424    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:44:49.345795 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338427    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:44:49.345795 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338430    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:44:49.345795 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338432    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:44:49.345795 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338435    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:44:49.345795 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338437    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:44:49.345795 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338440    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:44:49.345795 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338443    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:44:49.345795 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338445    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:44:49.346360 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338448    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:44:49.346360 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338456    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:44:49.346360 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338459    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:44:49.346360 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338462    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:44:49.346360 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338465    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:44:49.346360 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338468    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:44:49.346360 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338470    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:44:49.346360 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338473    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:44:49.346360 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338475    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:44:49.346360 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338478    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:44:49.346360 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338480    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:44:49.346360 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338483    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:44:49.346360 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338486    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:44:49.346360 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338488    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:44:49.346360 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338491    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:44:49.346360 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338493    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:44:49.346360 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338496    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:44:49.346360 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338499    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:44:49.346360 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338502    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:44:49.346360 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338505    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:44:49.346360 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338507    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:44:49.346988 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338510    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:44:49.346988 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338512    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:44:49.346988 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338515    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:44:49.346988 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338518    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:44:49.346988 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338520    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:44:49.346988 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338523    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:44:49.346988 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338525    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:44:49.346988 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338528    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:44:49.346988 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338531    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:44:49.346988 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338533    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:44:49.346988 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338537    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:44:49.346988 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338540    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:44:49.346988 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338544    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:44:49.346988 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338552    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:44:49.346988 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338555    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:44:49.346988 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338558    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:44:49.346988 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338560    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:44:49.346988 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338563    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:44:49.346988 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338565    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:44:49.347464 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338568    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:44:49.347464 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338570    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:44:49.347464 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338573    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:44:49.347464 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338575    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:44:49.347464 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338578    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:44:49.347464 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338581    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:44:49.347464 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338583    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:44:49.347464 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338586    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:44:49.347464 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338588    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:44:49.347464 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338591    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:44:49.347464 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338593    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:44:49.347464 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338596    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:44:49.347464 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338599    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:44:49.347464 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338601    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:44:49.347464 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338604    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:44:49.347464 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338623    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:44:49.347464 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338629    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:44:49.347464 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338635    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:44:49.347464 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338639    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:44:49.347958 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338643    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:44:49.347958 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338647    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:44:49.347958 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338650    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:44:49.347958 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338653    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:44:49.347958 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338656    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:44:49.347958 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338658    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:44:49.347958 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338661    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:44:49.347958 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338663    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:44:49.347958 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338673    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:44:49.347958 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338676    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:44:49.347958 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338678    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:44:49.347958 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338681    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:44:49.347958 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338684    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:44:49.347958 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338686    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:44:49.347958 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338689    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:44:49.347958 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338692    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:44:49.347958 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338694    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:44:49.347958 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338697    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:44:49.347958 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338699    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:44:49.347958 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338702    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:44:49.349276 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338704    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:44:49.349276 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338707    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:44:49.349276 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338710    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:44:49.349276 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338712    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:44:49.349276 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338715    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:44:49.349276 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338718    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:44:49.349276 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338721    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:44:49.349276 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338723    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:44:49.349276 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338726    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:44:49.349276 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338728    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:44:49.349276 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338731    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:44:49.349276 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338733    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:44:49.349276 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338736    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:44:49.349276 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.338760    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 17:44:49.349276 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338807    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:44:49.349276 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338812    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:44:49.349667 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338815    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:44:49.349667 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338818    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:44:49.349667 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338821    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:44:49.349667 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338824    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:44:49.349667 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338827    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:44:49.349667 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338835    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:44:49.349667 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338838    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:44:49.349667 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338840    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:44:49.349667 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338843    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:44:49.349667 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338845    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:44:49.349667 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338848    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:44:49.349667 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338851    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:44:49.349667 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338853    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:44:49.349667 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338856    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:44:49.349667 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338858    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:44:49.349667 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338861    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:44:49.349667 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338864    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:44:49.349667 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338866    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:44:49.349667 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338869    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:44:49.349667 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338871    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:44:49.350136 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338874    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:44:49.350136 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338876    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:44:49.350136 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338879    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:44:49.350136 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338882    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:44:49.350136 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338884    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:44:49.350136 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338904    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:44:49.350136 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338908    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:44:49.350136 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338911    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:44:49.350136 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338914    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:44:49.350136 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338916    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:44:49.350136 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338918    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:44:49.350136 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338921    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:44:49.350136 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338924    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:44:49.350136 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338927    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:44:49.350136 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338929    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:44:49.350136 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338932    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:44:49.350136 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338935    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:44:49.350136 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338937    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:44:49.350136 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338946    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:44:49.350136 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338948    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:44:49.350600 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338951    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:44:49.350600 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338954    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:44:49.350600 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338956    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:44:49.350600 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338959    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:44:49.350600 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338962    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:44:49.350600 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338964    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:44:49.350600 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338967    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:44:49.350600 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338969    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:44:49.350600 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338972    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:44:49.350600 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338975    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:44:49.350600 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338977    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:44:49.350600 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338980    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:44:49.350600 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338982    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:44:49.350600 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338985    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:44:49.350600 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338987    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:44:49.350600 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338991    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:44:49.350600 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338995    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:44:49.350600 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.338998    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:44:49.350600 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.339000    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:44:49.351088 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.339003    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:44:49.351088 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.339005    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:44:49.351088 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.339008    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:44:49.351088 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.339010    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:44:49.351088 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.339021    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:44:49.351088 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.339025    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:44:49.351088 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.339028    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:44:49.351088 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.339031    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:44:49.351088 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.339034    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:44:49.351088 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.339036    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:44:49.351088 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.339039    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:44:49.351088 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.339041    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:44:49.351088 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.339049    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:44:49.351088 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.339052    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:44:49.351088 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.339055    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:44:49.351088 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.339057    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:44:49.351088 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.339060    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:44:49.351088 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.339063    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:44:49.351088 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.339065    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:44:49.351526 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.339068    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:44:49.351526 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.339072    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:44:49.351526 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.339076    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:44:49.351526 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.339079    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:44:49.351526 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.339082    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:44:49.351526 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.339084    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:44:49.351526 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.339087    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:44:49.351526 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.339094    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:44:49.351526 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.339097    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:44:49.351526 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.339099    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:44:49.351526 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.339102    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:44:49.351526 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.339104    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:44:49.351526 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.339107    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:44:49.351526 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.339109    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:44:49.351526 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.339115    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:44:49.351526 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.339118    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:44:49.351526 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.339121    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:44:49.351526 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.339124    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:44:49.351526 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.339126    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:44:49.351526 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.339129    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:44:49.351999 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.339131    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:44:49.351999 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.339134    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:44:49.351999 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.339136    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:44:49.351999 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.339139    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:44:49.351999 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.339142    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:44:49.351999 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:49.339144    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:44:49.351999 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.339208    2576 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 17:44:49.351999 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.341157    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 03 17:44:49.351999 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.341973    2576 server.go:1014] "Starting client certificate rotation"
Nov 03 17:44:49.351999 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.342077    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 17:44:49.351999 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.342827    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 17:44:49.351999 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.343769    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 17:44:49.351999 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.347445    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 17:44:49.358900 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.358885    2576 log.go:25] "Validated CRI v1 runtime API"
Nov 03 17:44:49.361529 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.361509    2576 log.go:25] "Validated CRI v1 image API"
Nov 03 17:44:49.362375 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.362347    2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 17:44:49.365809 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.365790    2576 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8d4c12ac-84d9-41a3-8c88-9697de6cbccd:/dev/nvme0n1p3 d94ab8a4-0625-4ec3-a82f-ac022d9cf853:/dev/nvme0n1p4]
Nov 03 17:44:49.365859 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.365808    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 03 17:44:49.370094 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.369983    2576 manager.go:217] Machine: {Timestamp:2026-11-03 17:44:49.369504201 +0000 UTC m=+0.345151936 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3105081 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25fb36af538e17c812be0d5d125f23 SystemUUID:ec25fb36-af53-8e17-c812-be0d5d125f23 BootID:1fe403c3-07d4-45e7-8194-0505140c7d13 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:c5:e1:7a:8b Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:c5:e1:7a:8b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:aa:e4:c3:dd:f3:b0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 17:44:49.370555 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.370545    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 17:44:49.370656 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.370643    2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 17:44:49.371716 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.371696    2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 17:44:49.371856 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.371718    2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-204.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 17:44:49.371898 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.371857    2576 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 17:44:49.371898 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.371866    2576 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 17:44:49.371898 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.371876    2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 17:44:49.371898 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.371897    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 17:44:49.372622 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.372601    2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 17:44:49.375083 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.375072    2576 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 17:44:49.375218 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.375209    2576 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 17:44:49.375913 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.375904    2576 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 17:44:49.375944 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.375918    2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 17:44:49.375944 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.375929    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 17:44:49.375944 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.375938    2576 kubelet.go:405] "Adding apiserver pod source"
Nov 03 17:44:49.376028 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.375950    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 17:44:49.376286 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.376267    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 17:44:49.378842 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.378824    2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 17:44:49.382743 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.382731    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 17:44:49.383481 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.383469    2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 17:44:49.383521 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.383503    2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 17:44:49.384011 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.384000    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 17:44:49.384047 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.384018    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 17:44:49.384047 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.384025    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 17:44:49.384047 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.384030    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 17:44:49.384047 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.384035    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 17:44:49.384047 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.384041    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 17:44:49.384468 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.384460    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 17:44:49.384502 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.384469    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 17:44:49.384502 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.384476    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 17:44:49.384502 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.384482    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 17:44:49.384502 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.384490    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 17:44:49.384502 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.384499    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 17:44:49.385691 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.385681    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 17:44:49.385726 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.385695    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 17:44:49.389388 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.389376    2576 server.go:1263] "Started kubelet"
Nov 03 17:44:49.389487 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.389460    2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 17:44:49.392835 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.389487    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 17:44:49.392965 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.392950    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 17:44:49.393594 ip-10-0-128-204 systemd[1]: Started Kubernetes Kubelet.
Nov 03 17:44:49.395467 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.394878    2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 17:44:49.397785 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.397762    2576 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 17:44:49.398938 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:49.398896    2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 03 17:44:49.399638 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.399605    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 17:44:49.399999 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.399980    2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 17:44:49.400165 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:49.400136    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-204.us-west-2.compute.internal\" not found"
Nov 03 17:44:49.400534 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.400517    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 17:44:49.400631 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.400534    2576 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 17:44:49.400631 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.400550    2576 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 17:44:49.401816 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.400633    2576 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 17:44:49.401816 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.400644    2576 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 17:44:49.401816 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.400959    2576 factory.go:55] Registering systemd factory
Nov 03 17:44:49.401816 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.400974    2576 factory.go:223] Registration of the systemd container factory successfully
Nov 03 17:44:49.401816 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.401168    2576 factory.go:153] Registering CRI-O factory
Nov 03 17:44:49.401816 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.401179    2576 factory.go:223] Registration of the crio container factory successfully
Nov 03 17:44:49.401816 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.401223    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 17:44:49.401816 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.401242    2576 factory.go:103] Registering Raw factory
Nov 03 17:44:49.401816 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.401261    2576 manager.go:1199] Started watching for new ooms in manager
Nov 03 17:44:49.402188 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.402171    2576 manager.go:319] Starting recovery of all containers
Nov 03 17:44:49.413070 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.413050    2576 manager.go:324] Recovery completed
Nov 03 17:44:49.419130 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.419112    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:44:49.420927 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.420907    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-204.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:44:49.421013 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.420936    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-204.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:44:49.421013 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.420946    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-204.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:44:49.421504 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.421486    2576 cpu_manager.go:226] "Starting" policy="none"
Nov 03 17:44:49.421504 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.421502    2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 17:44:49.421637 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.421525    2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 17:44:49.421637 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.421506    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:44:49.422362 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.422346    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-204.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:44:49.422445 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.422376    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-204.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:44:49.422445 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.422390    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-204.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:44:49.422622 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.422596    2576 policy_none.go:50] "Start"
Nov 03 17:44:49.422680 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.422630    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 17:44:49.422680 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.422646    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 17:44:49.423757 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.423680    2576 policy_none.go:44] "Start"
Nov 03 17:44:49.453255 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.453215    2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 17:44:49.457579 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.457561    2576 manager.go:341] "Starting Device Plugin manager"
Nov 03 17:44:49.457676 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:49.457632    2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 17:44:49.457676 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.457644    2576 server.go:87] "Starting device plugin registration server"
Nov 03 17:44:49.457857 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.457845    2576 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 17:44:49.457914 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.457856    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 17:44:49.458011 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.457995    2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 17:44:49.458102 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.458085    2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 17:44:49.458179 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.458103    2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 17:44:49.458500 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:49.458481    2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 17:44:49.458566 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:49.458538    2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-204.us-west-2.compute.internal\" not found"
Nov 03 17:44:49.542567 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:49.542533    2576 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-204.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 03 17:44:49.544637 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.542602    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-204.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 03 17:44:49.544637 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:49.542534    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-204.us-west-2.compute.internal.18e49a0306c366e5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-204.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-204.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 17:44:49.389348581 +0000 UTC m=+0.364996316,LastTimestamp:2026-11-03 17:44:49.389348581 +0000 UTC m=+0.364996316,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-204.us-west-2.compute.internal,}"
Nov 03 17:44:49.554130 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.554101    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 17:44:49.555274 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.555256    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 17:44:49.555274 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.555275    2576 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 17:44:49.555424 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.555293    2576 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 17:44:49.555424 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:49.555322    2576 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 03 17:44:49.557935 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.557916    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:44:49.558748 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.558728    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-204.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:44:49.558831 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.558772    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-204.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:44:49.558831 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.558791    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-204.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:44:49.558831 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.558825    2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-204.us-west-2.compute.internal"
Nov 03 17:44:49.632744 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.632658    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 17:44:49.632897 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.632745    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-45wdr"
Nov 03 17:44:49.655751 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.655713    2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-204.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-204.us-west-2.compute.internal"]
Nov 03 17:44:49.655885 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.655785    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:44:49.656750 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.656734    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-204.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:44:49.656813 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.656765    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-204.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:44:49.656813 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.656776    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-204.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:44:49.658944 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.658932    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:44:49.659094 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.659077    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-204.us-west-2.compute.internal"
Nov 03 17:44:49.659175 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.659118    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:44:49.659619 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.659595    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-204.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:44:49.659686 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.659639    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-204.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:44:49.659686 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.659650    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-204.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:44:49.659776 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.659595    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-204.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:44:49.659776 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.659718    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-204.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:44:49.659776 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.659729    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-204.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:44:49.661744 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.661729    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-204.us-west-2.compute.internal"
Nov 03 17:44:49.661792 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.661754    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:44:49.662452 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.662436    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-204.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:44:49.662546 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.662468    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-204.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:44:49.662546 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.662483    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-204.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:44:49.690275 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:49.690070    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-204.us-west-2.compute.internal\" not found" node="ip-10-0-128-204.us-west-2.compute.internal"
Nov 03 17:44:49.692080 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:49.692065    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-204.us-west-2.compute.internal\" not found" node="ip-10-0-128-204.us-west-2.compute.internal"
Nov 03 17:44:49.695506 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.695491    2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:44:49.701488 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.701471    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8a90b512115a10076b69387005e99589-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-204.us-west-2.compute.internal\" (UID: \"8a90b512115a10076b69387005e99589\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-204.us-west-2.compute.internal"
Nov 03 17:44:49.701559 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.701502    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a90b512115a10076b69387005e99589-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-204.us-west-2.compute.internal\" (UID: \"8a90b512115a10076b69387005e99589\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-204.us-west-2.compute.internal"
Nov 03 17:44:49.701559 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.701517    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8dbd524262c9b524921d913509ef36fa-config\") pod \"kube-apiserver-proxy-ip-10-0-128-204.us-west-2.compute.internal\" (UID: \"8dbd524262c9b524921d913509ef36fa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-204.us-west-2.compute.internal"
Nov 03 17:44:49.740714 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.740687    2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:44:49.788201 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.788173    2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:44:49.788840 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.788808    2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:44:49.789631 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.789333    2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-204.us-west-2.compute.internal"
Nov 03 17:44:49.800509 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.800488    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-204.us-west-2.compute.internal"
Nov 03 17:44:49.801653 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.801635    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a90b512115a10076b69387005e99589-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-204.us-west-2.compute.internal\" (UID: \"8a90b512115a10076b69387005e99589\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-204.us-west-2.compute.internal"
Nov 03 17:44:49.801737 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.801662    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8dbd524262c9b524921d913509ef36fa-config\") pod \"kube-apiserver-proxy-ip-10-0-128-204.us-west-2.compute.internal\" (UID: \"8dbd524262c9b524921d913509ef36fa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-204.us-west-2.compute.internal"
Nov 03 17:44:49.801737 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.801678    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8a90b512115a10076b69387005e99589-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-204.us-west-2.compute.internal\" (UID: \"8a90b512115a10076b69387005e99589\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-204.us-west-2.compute.internal"
Nov 03 17:44:49.801737 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.801721    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8a90b512115a10076b69387005e99589-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-204.us-west-2.compute.internal\" (UID: \"8a90b512115a10076b69387005e99589\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-204.us-west-2.compute.internal"
Nov 03 17:44:49.801860 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.801742    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a90b512115a10076b69387005e99589-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-204.us-west-2.compute.internal\" (UID: \"8a90b512115a10076b69387005e99589\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-204.us-west-2.compute.internal"
Nov 03 17:44:49.801860 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.801746    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8dbd524262c9b524921d913509ef36fa-config\") pod \"kube-apiserver-proxy-ip-10-0-128-204.us-west-2.compute.internal\" (UID: \"8dbd524262c9b524921d913509ef36fa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-204.us-west-2.compute.internal"
Nov 03 17:44:49.860773 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.860750    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 17:44:49.860905 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.860862    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-204.us-west-2.compute.internal"
Nov 03 17:44:49.914321 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.914249    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 17:44:49.992938 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.992904    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-204.us-west-2.compute.internal"
Nov 03 17:44:49.996064 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:49.996041    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-204.us-west-2.compute.internal"
Nov 03 17:44:50.342420 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.342392    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 03 17:44:50.342888 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.342532    2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 17:44:50.342888 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.342562    2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 17:44:50.342888 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.342581    2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 17:44:50.342888 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:50.342630    2576 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://ad185ca6b177f4c7286538684ef3f4ce-d7e05c9bd6f41af3.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-128-204.us-west-2.compute.internal.18e49a0308a53ceb\": read tcp 10.0.128.204:41292->77.112.92.116:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-128-204.us-west-2.compute.internal.18e49a0308a53ceb  default   5142 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-204.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-204.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-204.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 17:44:49 +0000 UTC,LastTimestamp:2026-11-03 17:44:49.656752141 +0000 UTC m=+0.632399879,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-204.us-west-2.compute.internal,}"
Nov 03 17:44:50.376174 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.376151    2576 apiserver.go:52] "Watching apiserver"
Nov 03 17:44:50.400433 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.400414    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 03 17:44:50.637024 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.636908    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 17:39:49 +0000 UTC" deadline="2028-06-29 22:24:51.212838329 +0000 UTC"
Nov 03 17:44:50.637024 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.636962    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14500h40m0.575881303s"
Nov 03 17:44:50.637231 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.637177    2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 17:44:50.637505 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.637489    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-fjrg4","openshift-network-operator/iptables-alerter-n9qsq","openshift-multus/multus-additional-cni-plugins-bsgc5","openshift-multus/multus-jbbwp","kube-system/global-pull-secret-syncer-4xjw2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-204.us-west-2.compute.internal","openshift-multus/network-metrics-daemon-z264t","openshift-ovn-kubernetes/ovnkube-node-97p65","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6bks","openshift-dns/node-resolver-9j547","openshift-image-registry/node-ca-x8f2p","openshift-network-diagnostics/network-check-target-2t57w","kube-system/konnectivity-agent-dv4zl","kube-system/kube-apiserver-proxy-ip-10-0-128-204.us-west-2.compute.internal"]
Nov 03 17:44:50.639882 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.639868    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:44:50.641840 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.641824    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-n9qsq"
Nov 03 17:44:50.643838 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.643827    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bsgc5"
Nov 03 17:44:50.645671 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.645659    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jbbwp"
Nov 03 17:44:50.647639 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.647628    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xjw2"
Nov 03 17:44:50.647698 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:50.647683    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xjw2" podUID="f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30"
Nov 03 17:44:50.649604 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.649583    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z264t"
Nov 03 17:44:50.649677 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:50.649661    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z264t" podUID="9cf65c17-1c79-4cd5-9ef1-dceb467878b0"
Nov 03 17:44:50.651665 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.651644    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.653909 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.653891    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6bks"
Nov 03 17:44:50.656209 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.656190    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9j547"
Nov 03 17:44:50.658667 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.658646    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-x8f2p"
Nov 03 17:44:50.661803 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.661782    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2t57w"
Nov 03 17:44:50.661945 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:50.661922    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2t57w" podUID="43c9da28-334d-4bb3-a6bc-e3da2d561633"
Nov 03 17:44:50.663977 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.663962    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dv4zl"
Nov 03 17:44:50.689246 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.689215    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 17:44:50.689366 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.689257    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 17:44:50.689366 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.689300    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-m6lsw\""
Nov 03 17:44:50.691130 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.691110    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 17:44:50.691130 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.691123    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8lnxv\""
Nov 03 17:44:50.691241 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.691125    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 17:44:50.691241 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.691154    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 17:44:50.693048 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.693031    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 17:44:50.693152 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.693142    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 17:44:50.693477 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.693461    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 17:44:50.693545 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.693497    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 17:44:50.693545 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.693532    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 17:44:50.693645 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.693593    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5hxz7\""
Nov 03 17:44:50.694509 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.694495    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 17:44:50.694653 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.694635    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-j4vc9\""
Nov 03 17:44:50.701440 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.701422    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 17:44:50.707161 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.707138    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-host-run-ovn-kubernetes\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.707268 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.707165    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d59f146b-82ee-4f26-854a-cba274a89600-sys-fs\") pod \"aws-ebs-csi-driver-node-v6bks\" (UID: \"d59f146b-82ee-4f26-854a-cba274a89600\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6bks"
Nov 03 17:44:50.707268 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.707181    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b993b313-db8b-42f5-ad99-2d2ab5bb49e0-cnibin\") pod \"multus-additional-cni-plugins-bsgc5\" (UID: \"b993b313-db8b-42f5-ad99-2d2ab5bb49e0\") " pod="openshift-multus/multus-additional-cni-plugins-bsgc5"
Nov 03 17:44:50.707268 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.707202    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-host-var-lib-kubelet\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:44:50.707268 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.707249    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v89s6\" (UniqueName: \"kubernetes.io/projected/43c9da28-334d-4bb3-a6bc-e3da2d561633-kube-api-access-v89s6\") pod \"network-check-target-2t57w\" (UID: \"43c9da28-334d-4bb3-a6bc-e3da2d561633\") " pod="openshift-network-diagnostics/network-check-target-2t57w"
Nov 03 17:44:50.707392 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.707270    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-host-run-netns\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.707392 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.707316    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-log-socket\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.707392 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.707350    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d59f146b-82ee-4f26-854a-cba274a89600-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v6bks\" (UID: \"d59f146b-82ee-4f26-854a-cba274a89600\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6bks"
Nov 03 17:44:50.707392 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.707385    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b91c00dc-c14f-4b3a-b268-1eb6dcf9ad4b-konnectivity-ca\") pod \"konnectivity-agent-dv4zl\" (UID: \"b91c00dc-c14f-4b3a-b268-1eb6dcf9ad4b\") " pod="kube-system/konnectivity-agent-dv4zl"
Nov 03 17:44:50.707496 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.707417    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b993b313-db8b-42f5-ad99-2d2ab5bb49e0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bsgc5\" (UID: \"b993b313-db8b-42f5-ad99-2d2ab5bb49e0\") " pod="openshift-multus/multus-additional-cni-plugins-bsgc5"
Nov 03 17:44:50.707496 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.707441    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-host-run-k8s-cni-cncf-io\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:44:50.707496 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.707460    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-host-run-netns\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:44:50.707496 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.707473    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-host-cni-bin\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.707627 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.707501    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c29d200a-c917-46e5-9cf5-21f4d952e470-env-overrides\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.707627 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.707519    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq9fz\" (UniqueName: \"kubernetes.io/projected/b993b313-db8b-42f5-ad99-2d2ab5bb49e0-kube-api-access-kq9fz\") pod \"multus-additional-cni-plugins-bsgc5\" (UID: \"b993b313-db8b-42f5-ad99-2d2ab5bb49e0\") " pod="openshift-multus/multus-additional-cni-plugins-bsgc5"
Nov 03 17:44:50.707627 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.707545    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-multus-socket-dir-parent\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:44:50.707627 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.707565    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-multus-conf-dir\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:44:50.707627 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.707580    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-sys\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:44:50.707627 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.707593    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b993b313-db8b-42f5-ad99-2d2ab5bb49e0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bsgc5\" (UID: \"b993b313-db8b-42f5-ad99-2d2ab5bb49e0\") " pod="openshift-multus/multus-additional-cni-plugins-bsgc5"
Nov 03 17:44:50.707782 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.707629    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-system-cni-dir\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:44:50.707782 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.707660    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c29d200a-c917-46e5-9cf5-21f4d952e470-ovnkube-config\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.707782 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.707677    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b993b313-db8b-42f5-ad99-2d2ab5bb49e0-system-cni-dir\") pod \"multus-additional-cni-plugins-bsgc5\" (UID: \"b993b313-db8b-42f5-ad99-2d2ab5bb49e0\") " pod="openshift-multus/multus-additional-cni-plugins-bsgc5"
Nov 03 17:44:50.707782 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.707692    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-host-run-multus-certs\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:44:50.707782 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.707717    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plbfz\" (UniqueName: \"kubernetes.io/projected/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-kube-api-access-plbfz\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:44:50.707782 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.707738    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-run-systemd\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.707782 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.707752    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-run-openvswitch\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.707956 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.707784    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c29d200a-c917-46e5-9cf5-21f4d952e470-ovnkube-script-lib\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.707956 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.707807    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-etc-sysctl-conf\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:44:50.707956 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.707820    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-cni-binary-copy\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:44:50.707956 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.707840    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/79212f04-d10c-43ee-9dd5-41b4fd1917b2-serviceca\") pod \"node-ca-x8f2p\" (UID: \"79212f04-d10c-43ee-9dd5-41b4fd1917b2\") " pod="openshift-image-registry/node-ca-x8f2p"
Nov 03 17:44:50.707956 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.707862    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/778383aa-b3db-45a6-8bec-9891f7818e94-iptables-alerter-script\") pod \"iptables-alerter-n9qsq\" (UID: \"778383aa-b3db-45a6-8bec-9891f7818e94\") " pod="openshift-network-operator/iptables-alerter-n9qsq"
Nov 03 17:44:50.707956 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.707894    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-host-kubelet\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.707956 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.707938    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jssl5\" (UniqueName: \"kubernetes.io/projected/c29d200a-c917-46e5-9cf5-21f4d952e470-kube-api-access-jssl5\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.708130 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.707960    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d59f146b-82ee-4f26-854a-cba274a89600-etc-selinux\") pod \"aws-ebs-csi-driver-node-v6bks\" (UID: \"d59f146b-82ee-4f26-854a-cba274a89600\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6bks"
Nov 03 17:44:50.708130 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.707976    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-host\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:44:50.708130 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.707990    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b993b313-db8b-42f5-ad99-2d2ab5bb49e0-cni-binary-copy\") pod \"multus-additional-cni-plugins-bsgc5\" (UID: \"b993b313-db8b-42f5-ad99-2d2ab5bb49e0\") " pod="openshift-multus/multus-additional-cni-plugins-bsgc5"
Nov 03 17:44:50.708130 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.708008    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cf65c17-1c79-4cd5-9ef1-dceb467878b0-metrics-certs\") pod \"network-metrics-daemon-z264t\" (UID: \"9cf65c17-1c79-4cd5-9ef1-dceb467878b0\") " pod="openshift-multus/network-metrics-daemon-z264t"
Nov 03 17:44:50.708130 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.708020    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30-dbus\") pod \"global-pull-secret-syncer-4xjw2\" (UID: \"f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30\") " pod="kube-system/global-pull-secret-syncer-4xjw2"
Nov 03 17:44:50.708130 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.708033    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfmqk\" (UniqueName: \"kubernetes.io/projected/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-kube-api-access-rfmqk\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:44:50.708130 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.708046    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-multus-cni-dir\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:44:50.708130 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.708067    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-host-var-lib-cni-bin\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:44:50.708130 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.708104    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a831ae97-8f46-4b85-bfdb-59042b7b8b0c-tmp-dir\") pod \"node-resolver-9j547\" (UID: \"a831ae97-8f46-4b85-bfdb-59042b7b8b0c\") " pod="openshift-dns/node-resolver-9j547"
Nov 03 17:44:50.708473 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.708131    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-run\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:44:50.708473 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.708152    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djdsz\" (UniqueName: \"kubernetes.io/projected/79212f04-d10c-43ee-9dd5-41b4fd1917b2-kube-api-access-djdsz\") pod \"node-ca-x8f2p\" (UID: \"79212f04-d10c-43ee-9dd5-41b4fd1917b2\") " pod="openshift-image-registry/node-ca-x8f2p"
Nov 03 17:44:50.708473 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.708177    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-systemd-units\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.708473 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.708198    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-run-ovn\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.708473 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.708241    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-etc-kubernetes\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:44:50.708473 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.708274    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-etc-kubernetes\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:44:50.708473 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.708301    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtbgb\" (UniqueName: \"kubernetes.io/projected/9cf65c17-1c79-4cd5-9ef1-dceb467878b0-kube-api-access-vtbgb\") pod \"network-metrics-daemon-z264t\" (UID: \"9cf65c17-1c79-4cd5-9ef1-dceb467878b0\") " pod="openshift-multus/network-metrics-daemon-z264t"
Nov 03 17:44:50.708473 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.708328    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/778383aa-b3db-45a6-8bec-9891f7818e94-host-slash\") pod \"iptables-alerter-n9qsq\" (UID: \"778383aa-b3db-45a6-8bec-9891f7818e94\") " pod="openshift-network-operator/iptables-alerter-n9qsq"
Nov 03 17:44:50.708473 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.708349    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d59f146b-82ee-4f26-854a-cba274a89600-socket-dir\") pod \"aws-ebs-csi-driver-node-v6bks\" (UID: \"d59f146b-82ee-4f26-854a-cba274a89600\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6bks"
Nov 03 17:44:50.708473 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.708382    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b91c00dc-c14f-4b3a-b268-1eb6dcf9ad4b-agent-certs\") pod \"konnectivity-agent-dv4zl\" (UID: \"b91c00dc-c14f-4b3a-b268-1eb6dcf9ad4b\") " pod="kube-system/konnectivity-agent-dv4zl"
Nov 03 17:44:50.708473 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.708418    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-etc-modprobe-d\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:44:50.708473 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.708445    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-lib-modules\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:44:50.708473 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.708470    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-var-lib-kubelet\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:44:50.708831 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.708487    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-multus-daemon-config\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:44:50.708831 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.708517    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30-original-pull-secret\") pod \"global-pull-secret-syncer-4xjw2\" (UID: \"f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30\") " pod="kube-system/global-pull-secret-syncer-4xjw2"
Nov 03 17:44:50.708831 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.708537    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-host-slash\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.708831 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.708552    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzjx9\" (UniqueName: \"kubernetes.io/projected/d59f146b-82ee-4f26-854a-cba274a89600-kube-api-access-bzjx9\") pod \"aws-ebs-csi-driver-node-v6bks\" (UID: \"d59f146b-82ee-4f26-854a-cba274a89600\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6bks"
Nov 03 17:44:50.708831 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.708585    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79212f04-d10c-43ee-9dd5-41b4fd1917b2-host\") pod \"node-ca-x8f2p\" (UID: \"79212f04-d10c-43ee-9dd5-41b4fd1917b2\") " pod="openshift-image-registry/node-ca-x8f2p"
Nov 03 17:44:50.708831 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.708637    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-node-log\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.708831 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.708663    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c29d200a-c917-46e5-9cf5-21f4d952e470-ovn-node-metrics-cert\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.708831 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.708689    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-etc-sysconfig\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:44:50.708831 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.708708    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-etc-systemd\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:44:50.708831 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.708725    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-etc-tuned\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:44:50.708831 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.708749    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpp49\" (UniqueName: \"kubernetes.io/projected/a831ae97-8f46-4b85-bfdb-59042b7b8b0c-kube-api-access-mpp49\") pod \"node-resolver-9j547\" (UID: \"a831ae97-8f46-4b85-bfdb-59042b7b8b0c\") " pod="openshift-dns/node-resolver-9j547"
Nov 03 17:44:50.708831 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.708771    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.708831 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.708788    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-etc-sysctl-d\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:44:50.708831 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.708804    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a831ae97-8f46-4b85-bfdb-59042b7b8b0c-hosts-file\") pod \"node-resolver-9j547\" (UID: \"a831ae97-8f46-4b85-bfdb-59042b7b8b0c\") " pod="openshift-dns/node-resolver-9j547"
Nov 03 17:44:50.708831 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.708832    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt25m\" (UniqueName: \"kubernetes.io/projected/778383aa-b3db-45a6-8bec-9891f7818e94-kube-api-access-dt25m\") pod \"iptables-alerter-n9qsq\" (UID: \"778383aa-b3db-45a6-8bec-9891f7818e94\") " pod="openshift-network-operator/iptables-alerter-n9qsq"
Nov 03 17:44:50.709332 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.708849    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d59f146b-82ee-4f26-854a-cba274a89600-device-dir\") pod \"aws-ebs-csi-driver-node-v6bks\" (UID: \"d59f146b-82ee-4f26-854a-cba274a89600\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6bks"
Nov 03 17:44:50.709332 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.708866    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b993b313-db8b-42f5-ad99-2d2ab5bb49e0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bsgc5\" (UID: \"b993b313-db8b-42f5-ad99-2d2ab5bb49e0\") " pod="openshift-multus/multus-additional-cni-plugins-bsgc5"
Nov 03 17:44:50.709332 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.708880    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-host-var-lib-cni-multus\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:44:50.709332 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.708897    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-etc-openvswitch\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.709332 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.708915    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-host-cni-netd\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.709332 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.708929    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d59f146b-82ee-4f26-854a-cba274a89600-registration-dir\") pod \"aws-ebs-csi-driver-node-v6bks\" (UID: \"d59f146b-82ee-4f26-854a-cba274a89600\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6bks"
Nov 03 17:44:50.709332 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.708941    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-tmp\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:44:50.709332 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.708954    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-cnibin\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:44:50.709332 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.708987    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-hostroot\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:44:50.709332 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.709025    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30-kubelet-config\") pod \"global-pull-secret-syncer-4xjw2\" (UID: \"f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30\") " pod="kube-system/global-pull-secret-syncer-4xjw2"
Nov 03 17:44:50.709332 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.709072    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-var-lib-openvswitch\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.780543 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.778199    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 17:44:50.780543 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.778256    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-t4hth"
Nov 03 17:44:50.781038 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.781014    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 17:44:50.781159 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.781070    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 17:44:50.781245 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.781231    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 17:44:50.781329 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.781315    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 17:44:50.781383 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.781364    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 17:44:50.781443 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.781403    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-l7l2n\""
Nov 03 17:44:50.781443 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.781409    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 17:44:50.781443 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.781411    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 17:44:50.781443 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.781403    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 17:44:50.781595 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.781573    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 17:44:50.781694 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.781619    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mk8dz\""
Nov 03 17:44:50.781694 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.781627    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 17:44:50.781791 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.781699    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-z56rs\""
Nov 03 17:44:50.781791 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.781726    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 17:44:50.781791 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.781731    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 17:44:50.781791 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.781725    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 17:44:50.781791 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.781734    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-bmd8j\""
Nov 03 17:44:50.781791 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.781754    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 17:44:50.781791 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.781762    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 17:44:50.781791 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.781732    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 17:44:50.781791 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.781776    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8w5gn\""
Nov 03 17:44:50.809376 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.809354    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30-kubelet-config\") pod \"global-pull-secret-syncer-4xjw2\" (UID: \"f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30\") " pod="kube-system/global-pull-secret-syncer-4xjw2"
Nov 03 17:44:50.809376 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.809377    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-var-lib-openvswitch\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.809533 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.809391    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-host-run-ovn-kubernetes\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.809533 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.809405    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d59f146b-82ee-4f26-854a-cba274a89600-sys-fs\") pod \"aws-ebs-csi-driver-node-v6bks\" (UID: \"d59f146b-82ee-4f26-854a-cba274a89600\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6bks"
Nov 03 17:44:50.809533 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.809420    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b993b313-db8b-42f5-ad99-2d2ab5bb49e0-cnibin\") pod \"multus-additional-cni-plugins-bsgc5\" (UID: \"b993b313-db8b-42f5-ad99-2d2ab5bb49e0\") " pod="openshift-multus/multus-additional-cni-plugins-bsgc5"
Nov 03 17:44:50.809533 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.809436    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-host-var-lib-kubelet\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:44:50.809533 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.809468    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-var-lib-openvswitch\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.809533 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.809477    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-host-run-ovn-kubernetes\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.809533 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.809487    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b993b313-db8b-42f5-ad99-2d2ab5bb49e0-cnibin\") pod \"multus-additional-cni-plugins-bsgc5\" (UID: \"b993b313-db8b-42f5-ad99-2d2ab5bb49e0\") " pod="openshift-multus/multus-additional-cni-plugins-bsgc5"
Nov 03 17:44:50.809533 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.809486    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d59f146b-82ee-4f26-854a-cba274a89600-sys-fs\") pod \"aws-ebs-csi-driver-node-v6bks\" (UID: \"d59f146b-82ee-4f26-854a-cba274a89600\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6bks"
Nov 03 17:44:50.809533 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.809486    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-host-var-lib-kubelet\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:44:50.809533 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.809486    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v89s6\" (UniqueName: \"kubernetes.io/projected/43c9da28-334d-4bb3-a6bc-e3da2d561633-kube-api-access-v89s6\") pod \"network-check-target-2t57w\" (UID: \"43c9da28-334d-4bb3-a6bc-e3da2d561633\") " pod="openshift-network-diagnostics/network-check-target-2t57w"
Nov 03 17:44:50.809533 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.809526    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30-kubelet-config\") pod \"global-pull-secret-syncer-4xjw2\" (UID: \"f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30\") " pod="kube-system/global-pull-secret-syncer-4xjw2"
Nov 03 17:44:50.809999 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.809555    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-host-run-netns\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.809999 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.809572    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-log-socket\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.809999 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.809590    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-host-run-netns\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.809999 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.809594    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d59f146b-82ee-4f26-854a-cba274a89600-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v6bks\" (UID: \"d59f146b-82ee-4f26-854a-cba274a89600\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6bks"
Nov 03 17:44:50.809999 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.809637    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b91c00dc-c14f-4b3a-b268-1eb6dcf9ad4b-konnectivity-ca\") pod \"konnectivity-agent-dv4zl\" (UID: \"b91c00dc-c14f-4b3a-b268-1eb6dcf9ad4b\") " pod="kube-system/konnectivity-agent-dv4zl"
Nov 03 17:44:50.809999 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.809641    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d59f146b-82ee-4f26-854a-cba274a89600-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v6bks\" (UID: \"d59f146b-82ee-4f26-854a-cba274a89600\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6bks"
Nov 03 17:44:50.809999 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.809651    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b993b313-db8b-42f5-ad99-2d2ab5bb49e0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bsgc5\" (UID: \"b993b313-db8b-42f5-ad99-2d2ab5bb49e0\") " pod="openshift-multus/multus-additional-cni-plugins-bsgc5"
Nov 03 17:44:50.809999 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.809653    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-log-socket\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.809999 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.809674    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-host-run-k8s-cni-cncf-io\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:44:50.809999 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.809691    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-host-run-netns\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:44:50.809999 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.809703    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-host-cni-bin\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.809999 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.809716    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c29d200a-c917-46e5-9cf5-21f4d952e470-env-overrides\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.809999 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.809736    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq9fz\" (UniqueName: \"kubernetes.io/projected/b993b313-db8b-42f5-ad99-2d2ab5bb49e0-kube-api-access-kq9fz\") pod \"multus-additional-cni-plugins-bsgc5\" (UID: \"b993b313-db8b-42f5-ad99-2d2ab5bb49e0\") " pod="openshift-multus/multus-additional-cni-plugins-bsgc5"
Nov 03 17:44:50.809999 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.809755    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b993b313-db8b-42f5-ad99-2d2ab5bb49e0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bsgc5\" (UID: \"b993b313-db8b-42f5-ad99-2d2ab5bb49e0\") " pod="openshift-multus/multus-additional-cni-plugins-bsgc5"
Nov 03 17:44:50.809999 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.809757    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-host-run-netns\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:44:50.809999 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.809768    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-host-cni-bin\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.809999 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.809799    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-host-run-k8s-cni-cncf-io\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:44:50.810768 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.809830    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-multus-socket-dir-parent\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:44:50.810768 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.809852    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-multus-conf-dir\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:44:50.810768 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.809874    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-sys\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:44:50.810768 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.809895    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b993b313-db8b-42f5-ad99-2d2ab5bb49e0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bsgc5\" (UID: \"b993b313-db8b-42f5-ad99-2d2ab5bb49e0\") " pod="openshift-multus/multus-additional-cni-plugins-bsgc5"
Nov 03 17:44:50.810768 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.809919    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-system-cni-dir\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:44:50.810768 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.809928    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-multus-conf-dir\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:44:50.810768 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.809940    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c29d200a-c917-46e5-9cf5-21f4d952e470-ovnkube-config\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.810768 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.809961    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-multus-socket-dir-parent\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:44:50.810768 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.809967    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-sys\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:44:50.810768 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.809988    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-system-cni-dir\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:44:50.810768 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810013    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b993b313-db8b-42f5-ad99-2d2ab5bb49e0-system-cni-dir\") pod \"multus-additional-cni-plugins-bsgc5\" (UID: \"b993b313-db8b-42f5-ad99-2d2ab5bb49e0\") " pod="openshift-multus/multus-additional-cni-plugins-bsgc5"
Nov 03 17:44:50.810768 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810036    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-host-run-multus-certs\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:44:50.810768 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810056    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b993b313-db8b-42f5-ad99-2d2ab5bb49e0-system-cni-dir\") pod \"multus-additional-cni-plugins-bsgc5\" (UID: \"b993b313-db8b-42f5-ad99-2d2ab5bb49e0\") " pod="openshift-multus/multus-additional-cni-plugins-bsgc5"
Nov 03 17:44:50.810768 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810057    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plbfz\" (UniqueName: \"kubernetes.io/projected/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-kube-api-access-plbfz\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:44:50.810768 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810078    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-run-systemd\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.810768 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810122    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-run-openvswitch\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.810768 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810139    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c29d200a-c917-46e5-9cf5-21f4d952e470-ovnkube-script-lib\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.810768 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810152    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-etc-sysctl-conf\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:44:50.811623 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810156    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c29d200a-c917-46e5-9cf5-21f4d952e470-env-overrides\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.811623 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810173    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-cni-binary-copy\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:44:50.811623 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810191    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/79212f04-d10c-43ee-9dd5-41b4fd1917b2-serviceca\") pod \"node-ca-x8f2p\" (UID: \"79212f04-d10c-43ee-9dd5-41b4fd1917b2\") " pod="openshift-image-registry/node-ca-x8f2p"
Nov 03 17:44:50.811623 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810192    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b91c00dc-c14f-4b3a-b268-1eb6dcf9ad4b-konnectivity-ca\") pod \"konnectivity-agent-dv4zl\" (UID: \"b91c00dc-c14f-4b3a-b268-1eb6dcf9ad4b\") " pod="kube-system/konnectivity-agent-dv4zl"
Nov 03 17:44:50.811623 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810206    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/778383aa-b3db-45a6-8bec-9891f7818e94-iptables-alerter-script\") pod \"iptables-alerter-n9qsq\" (UID: \"778383aa-b3db-45a6-8bec-9891f7818e94\") " pod="openshift-network-operator/iptables-alerter-n9qsq"
Nov 03 17:44:50.811623 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810219    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-host-kubelet\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.811623 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810235    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jssl5\" (UniqueName: \"kubernetes.io/projected/c29d200a-c917-46e5-9cf5-21f4d952e470-kube-api-access-jssl5\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.811623 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810246    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-run-systemd\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.811623 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810254    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d59f146b-82ee-4f26-854a-cba274a89600-etc-selinux\") pod \"aws-ebs-csi-driver-node-v6bks\" (UID: \"d59f146b-82ee-4f26-854a-cba274a89600\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6bks"
Nov 03 17:44:50.811623 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810289    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-host\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:44:50.811623 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810313    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b993b313-db8b-42f5-ad99-2d2ab5bb49e0-cni-binary-copy\") pod \"multus-additional-cni-plugins-bsgc5\" (UID: \"b993b313-db8b-42f5-ad99-2d2ab5bb49e0\") " pod="openshift-multus/multus-additional-cni-plugins-bsgc5"
Nov 03 17:44:50.811623 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810331    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d59f146b-82ee-4f26-854a-cba274a89600-etc-selinux\") pod \"aws-ebs-csi-driver-node-v6bks\" (UID: \"d59f146b-82ee-4f26-854a-cba274a89600\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6bks"
Nov 03 17:44:50.811623 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810338    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cf65c17-1c79-4cd5-9ef1-dceb467878b0-metrics-certs\") pod \"network-metrics-daemon-z264t\" (UID: \"9cf65c17-1c79-4cd5-9ef1-dceb467878b0\") " pod="openshift-multus/network-metrics-daemon-z264t"
Nov 03 17:44:50.811623 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810359    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30-dbus\") pod \"global-pull-secret-syncer-4xjw2\" (UID: \"f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30\") " pod="kube-system/global-pull-secret-syncer-4xjw2"
Nov 03 17:44:50.811623 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810380    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfmqk\" (UniqueName: \"kubernetes.io/projected/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-kube-api-access-rfmqk\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:44:50.811623 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810386    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c29d200a-c917-46e5-9cf5-21f4d952e470-ovnkube-config\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.811623 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810403    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-multus-cni-dir\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:44:50.812277 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810425    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-host-var-lib-cni-bin\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:44:50.812277 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810434    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-host-kubelet\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.812277 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810443    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b993b313-db8b-42f5-ad99-2d2ab5bb49e0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bsgc5\" (UID: \"b993b313-db8b-42f5-ad99-2d2ab5bb49e0\") " pod="openshift-multus/multus-additional-cni-plugins-bsgc5"
Nov 03 17:44:50.812277 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810445    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a831ae97-8f46-4b85-bfdb-59042b7b8b0c-tmp-dir\") pod \"node-resolver-9j547\" (UID: \"a831ae97-8f46-4b85-bfdb-59042b7b8b0c\") " pod="openshift-dns/node-resolver-9j547"
Nov 03 17:44:50.812277 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810481    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-run\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:44:50.812277 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810507    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djdsz\" (UniqueName: \"kubernetes.io/projected/79212f04-d10c-43ee-9dd5-41b4fd1917b2-kube-api-access-djdsz\") pod \"node-ca-x8f2p\" (UID: \"79212f04-d10c-43ee-9dd5-41b4fd1917b2\") " pod="openshift-image-registry/node-ca-x8f2p"
Nov 03 17:44:50.812277 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810541    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-systemd-units\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.812277 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810564    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-run-ovn\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.812277 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810586    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-etc-kubernetes\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:44:50.812277 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810625    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-etc-kubernetes\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:44:50.812277 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810649    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtbgb\" (UniqueName: \"kubernetes.io/projected/9cf65c17-1c79-4cd5-9ef1-dceb467878b0-kube-api-access-vtbgb\") pod \"network-metrics-daemon-z264t\" (UID: \"9cf65c17-1c79-4cd5-9ef1-dceb467878b0\") " pod="openshift-multus/network-metrics-daemon-z264t"
Nov 03 17:44:50.812277 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810674    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/778383aa-b3db-45a6-8bec-9891f7818e94-host-slash\") pod \"iptables-alerter-n9qsq\" (UID: \"778383aa-b3db-45a6-8bec-9891f7818e94\") " pod="openshift-network-operator/iptables-alerter-n9qsq"
Nov 03 17:44:50.812277 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810694    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d59f146b-82ee-4f26-854a-cba274a89600-socket-dir\") pod \"aws-ebs-csi-driver-node-v6bks\" (UID: \"d59f146b-82ee-4f26-854a-cba274a89600\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6bks"
Nov 03 17:44:50.812277 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810696    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/79212f04-d10c-43ee-9dd5-41b4fd1917b2-serviceca\") pod \"node-ca-x8f2p\" (UID: \"79212f04-d10c-43ee-9dd5-41b4fd1917b2\") " pod="openshift-image-registry/node-ca-x8f2p"
Nov 03 17:44:50.812277 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810722    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b91c00dc-c14f-4b3a-b268-1eb6dcf9ad4b-agent-certs\") pod \"konnectivity-agent-dv4zl\" (UID: \"b91c00dc-c14f-4b3a-b268-1eb6dcf9ad4b\") " pod="kube-system/konnectivity-agent-dv4zl"
Nov 03 17:44:50.812277 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810730    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-host-run-multus-certs\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:44:50.812277 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810752    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-run\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:44:50.812277 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810760    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a831ae97-8f46-4b85-bfdb-59042b7b8b0c-tmp-dir\") pod \"node-resolver-9j547\" (UID: \"a831ae97-8f46-4b85-bfdb-59042b7b8b0c\") " pod="openshift-dns/node-resolver-9j547"
Nov 03 17:44:50.813036 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810803    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-systemd-units\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.813036 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810752    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-etc-modprobe-d\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:44:50.813036 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810832    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-lib-modules\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:44:50.813036 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810839    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-etc-modprobe-d\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:44:50.813036 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810853    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-var-lib-kubelet\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:44:50.813036 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810694    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c29d200a-c917-46e5-9cf5-21f4d952e470-ovnkube-script-lib\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.813036 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810872    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-etc-sysctl-conf\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:44:50.813036 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810877    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-multus-daemon-config\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:44:50.813036 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810881    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-run-openvswitch\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.813036 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810900    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30-original-pull-secret\") pod \"global-pull-secret-syncer-4xjw2\" (UID: \"f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30\") " pod="kube-system/global-pull-secret-syncer-4xjw2"
Nov 03 17:44:50.813036 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810945    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/778383aa-b3db-45a6-8bec-9891f7818e94-iptables-alerter-script\") pod \"iptables-alerter-n9qsq\" (UID: \"778383aa-b3db-45a6-8bec-9891f7818e94\") " pod="openshift-network-operator/iptables-alerter-n9qsq"
Nov 03 17:44:50.813036 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810957    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-etc-kubernetes\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:44:50.813036 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810960    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d59f146b-82ee-4f26-854a-cba274a89600-socket-dir\") pod \"aws-ebs-csi-driver-node-v6bks\" (UID: \"d59f146b-82ee-4f26-854a-cba274a89600\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6bks"
Nov 03 17:44:50.813036 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810763    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-cni-binary-copy\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:44:50.813036 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810990    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-lib-modules\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:44:50.813036 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810994    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30-dbus\") pod \"global-pull-secret-syncer-4xjw2\" (UID: \"f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30\") " pod="kube-system/global-pull-secret-syncer-4xjw2"
Nov 03 17:44:50.813036 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.810999    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-host-var-lib-cni-bin\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:44:50.813036 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.811012    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-host-slash\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.813837 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.811013    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-var-lib-kubelet\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:44:50.813837 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.811013    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-host\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:44:50.813837 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.811029    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzjx9\" (UniqueName: \"kubernetes.io/projected/d59f146b-82ee-4f26-854a-cba274a89600-kube-api-access-bzjx9\") pod \"aws-ebs-csi-driver-node-v6bks\" (UID: \"d59f146b-82ee-4f26-854a-cba274a89600\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6bks"
Nov 03 17:44:50.813837 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.811051    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-run-ovn\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.813837 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.811057    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-host-slash\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.813837 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.811068    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-multus-cni-dir\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:44:50.813837 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.811081    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-etc-kubernetes\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:44:50.813837 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.811104    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79212f04-d10c-43ee-9dd5-41b4fd1917b2-host\") pod \"node-ca-x8f2p\" (UID: \"79212f04-d10c-43ee-9dd5-41b4fd1917b2\") " pod="openshift-image-registry/node-ca-x8f2p"
Nov 03 17:44:50.813837 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.811114    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/778383aa-b3db-45a6-8bec-9891f7818e94-host-slash\") pod \"iptables-alerter-n9qsq\" (UID: \"778383aa-b3db-45a6-8bec-9891f7818e94\") " pod="openshift-network-operator/iptables-alerter-n9qsq"
Nov 03 17:44:50.813837 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.811127    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-node-log\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.813837 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:50.811132    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:44:50.813837 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:50.811154    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 17:44:50.813837 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.811237    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c29d200a-c917-46e5-9cf5-21f4d952e470-ovn-node-metrics-cert\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.813837 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.811249    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-node-log\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.813837 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.811251    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79212f04-d10c-43ee-9dd5-41b4fd1917b2-host\") pod \"node-ca-x8f2p\" (UID: \"79212f04-d10c-43ee-9dd5-41b4fd1917b2\") " pod="openshift-image-registry/node-ca-x8f2p"
Nov 03 17:44:50.813837 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.811262    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-etc-sysconfig\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:44:50.813837 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:50.811305    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cf65c17-1c79-4cd5-9ef1-dceb467878b0-metrics-certs podName:9cf65c17-1c79-4cd5-9ef1-dceb467878b0 nodeName:}" failed. No retries permitted until 2026-11-03 17:44:51.311266069 +0000 UTC m=+2.286913809 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9cf65c17-1c79-4cd5-9ef1-dceb467878b0-metrics-certs") pod "network-metrics-daemon-z264t" (UID: "9cf65c17-1c79-4cd5-9ef1-dceb467878b0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:44:50.814475 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:50.811351    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30-original-pull-secret podName:f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30 nodeName:}" failed. No retries permitted until 2026-11-03 17:44:51.311333786 +0000 UTC m=+2.286981521 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30-original-pull-secret") pod "global-pull-secret-syncer-4xjw2" (UID: "f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30") : object "kube-system"/"original-pull-secret" not registered
Nov 03 17:44:50.814475 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.811360    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-etc-sysconfig\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:44:50.814475 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.811372    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-etc-systemd\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:44:50.814475 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.811416    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-etc-tuned\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:44:50.814475 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.811417    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-etc-systemd\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:44:50.814475 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.811446    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpp49\" (UniqueName: \"kubernetes.io/projected/a831ae97-8f46-4b85-bfdb-59042b7b8b0c-kube-api-access-mpp49\") pod \"node-resolver-9j547\" (UID: \"a831ae97-8f46-4b85-bfdb-59042b7b8b0c\") " pod="openshift-dns/node-resolver-9j547"
Nov 03 17:44:50.814475 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.811467    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-multus-daemon-config\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:44:50.814475 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.811464    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b993b313-db8b-42f5-ad99-2d2ab5bb49e0-cni-binary-copy\") pod \"multus-additional-cni-plugins-bsgc5\" (UID: \"b993b313-db8b-42f5-ad99-2d2ab5bb49e0\") " pod="openshift-multus/multus-additional-cni-plugins-bsgc5"
Nov 03 17:44:50.814475 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.811483    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.814475 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.811526    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.814475 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.811536    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-etc-sysctl-d\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:44:50.814475 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.811562    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a831ae97-8f46-4b85-bfdb-59042b7b8b0c-hosts-file\") pod \"node-resolver-9j547\" (UID: \"a831ae97-8f46-4b85-bfdb-59042b7b8b0c\") " pod="openshift-dns/node-resolver-9j547"
Nov 03 17:44:50.814475 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.811577    2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 17:44:50.814475 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.811592    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt25m\" (UniqueName: \"kubernetes.io/projected/778383aa-b3db-45a6-8bec-9891f7818e94-kube-api-access-dt25m\") pod \"iptables-alerter-n9qsq\" (UID: \"778383aa-b3db-45a6-8bec-9891f7818e94\") " pod="openshift-network-operator/iptables-alerter-n9qsq"
Nov 03 17:44:50.814475 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.811635    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d59f146b-82ee-4f26-854a-cba274a89600-device-dir\") pod \"aws-ebs-csi-driver-node-v6bks\" (UID: \"d59f146b-82ee-4f26-854a-cba274a89600\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6bks"
Nov 03 17:44:50.814475 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.811645    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a831ae97-8f46-4b85-bfdb-59042b7b8b0c-hosts-file\") pod \"node-resolver-9j547\" (UID: \"a831ae97-8f46-4b85-bfdb-59042b7b8b0c\") " pod="openshift-dns/node-resolver-9j547"
Nov 03 17:44:50.814904 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.811658    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b993b313-db8b-42f5-ad99-2d2ab5bb49e0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bsgc5\" (UID: \"b993b313-db8b-42f5-ad99-2d2ab5bb49e0\") " pod="openshift-multus/multus-additional-cni-plugins-bsgc5"
Nov 03 17:44:50.814904 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.811670    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-etc-sysctl-d\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:44:50.814904 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.811679    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-host-var-lib-cni-multus\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:44:50.814904 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.811695    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-etc-openvswitch\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.814904 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.811710    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-host-cni-netd\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.814904 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.811727    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-host-var-lib-cni-multus\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:44:50.814904 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.811732    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d59f146b-82ee-4f26-854a-cba274a89600-registration-dir\") pod \"aws-ebs-csi-driver-node-v6bks\" (UID: \"d59f146b-82ee-4f26-854a-cba274a89600\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6bks"
Nov 03 17:44:50.814904 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.811753    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-etc-openvswitch\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.814904 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.811755    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-tmp\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:44:50.814904 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.811770    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-host-cni-netd\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.814904 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.811777    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d59f146b-82ee-4f26-854a-cba274a89600-registration-dir\") pod \"aws-ebs-csi-driver-node-v6bks\" (UID: \"d59f146b-82ee-4f26-854a-cba274a89600\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6bks"
Nov 03 17:44:50.814904 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.811692    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d59f146b-82ee-4f26-854a-cba274a89600-device-dir\") pod \"aws-ebs-csi-driver-node-v6bks\" (UID: \"d59f146b-82ee-4f26-854a-cba274a89600\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6bks"
Nov 03 17:44:50.814904 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.811788    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-cnibin\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:44:50.814904 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.811809    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-hostroot\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:44:50.814904 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.811865    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-hostroot\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:44:50.814904 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.811896    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-cnibin\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:44:50.814904 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.812166    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b993b313-db8b-42f5-ad99-2d2ab5bb49e0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bsgc5\" (UID: \"b993b313-db8b-42f5-ad99-2d2ab5bb49e0\") " pod="openshift-multus/multus-additional-cni-plugins-bsgc5"
Nov 03 17:44:50.815310 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.814535    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c29d200a-c917-46e5-9cf5-21f4d952e470-ovn-node-metrics-cert\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.815310 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.814550    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-tmp\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:44:50.815310 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.814623    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b91c00dc-c14f-4b3a-b268-1eb6dcf9ad4b-agent-certs\") pod \"konnectivity-agent-dv4zl\" (UID: \"b91c00dc-c14f-4b3a-b268-1eb6dcf9ad4b\") " pod="kube-system/konnectivity-agent-dv4zl"
Nov 03 17:44:50.815310 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.814691    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-etc-tuned\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:44:50.864449 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:50.864423    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 17:44:50.864449 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:50.864446    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 17:44:50.864449 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:50.864457    2576 projected.go:196] Error preparing data for projected volume kube-api-access-v89s6 for pod openshift-network-diagnostics/network-check-target-2t57w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:44:50.864679 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:50.864513    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43c9da28-334d-4bb3-a6bc-e3da2d561633-kube-api-access-v89s6 podName:43c9da28-334d-4bb3-a6bc-e3da2d561633 nodeName:}" failed. No retries permitted until 2026-11-03 17:44:51.364499138 +0000 UTC m=+2.340146870 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-v89s6" (UniqueName: "kubernetes.io/projected/43c9da28-334d-4bb3-a6bc-e3da2d561633-kube-api-access-v89s6") pod "network-check-target-2t57w" (UID: "43c9da28-334d-4bb3-a6bc-e3da2d561633") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:44:50.867260 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.867233    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plbfz\" (UniqueName: \"kubernetes.io/projected/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-kube-api-access-plbfz\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:44:50.867399 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.867356    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq9fz\" (UniqueName: \"kubernetes.io/projected/b993b313-db8b-42f5-ad99-2d2ab5bb49e0-kube-api-access-kq9fz\") pod \"multus-additional-cni-plugins-bsgc5\" (UID: \"b993b313-db8b-42f5-ad99-2d2ab5bb49e0\") " pod="openshift-multus/multus-additional-cni-plugins-bsgc5"
Nov 03 17:44:50.867541 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.867515    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jssl5\" (UniqueName: \"kubernetes.io/projected/c29d200a-c917-46e5-9cf5-21f4d952e470-kube-api-access-jssl5\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:50.867786 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.867762    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djdsz\" (UniqueName: \"kubernetes.io/projected/79212f04-d10c-43ee-9dd5-41b4fd1917b2-kube-api-access-djdsz\") pod \"node-ca-x8f2p\" (UID: \"79212f04-d10c-43ee-9dd5-41b4fd1917b2\") " pod="openshift-image-registry/node-ca-x8f2p"
Nov 03 17:44:50.871011 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.870992    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtbgb\" (UniqueName: \"kubernetes.io/projected/9cf65c17-1c79-4cd5-9ef1-dceb467878b0-kube-api-access-vtbgb\") pod \"network-metrics-daemon-z264t\" (UID: \"9cf65c17-1c79-4cd5-9ef1-dceb467878b0\") " pod="openshift-multus/network-metrics-daemon-z264t"
Nov 03 17:44:50.871396 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.871373    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt25m\" (UniqueName: \"kubernetes.io/projected/778383aa-b3db-45a6-8bec-9891f7818e94-kube-api-access-dt25m\") pod \"iptables-alerter-n9qsq\" (UID: \"778383aa-b3db-45a6-8bec-9891f7818e94\") " pod="openshift-network-operator/iptables-alerter-n9qsq"
Nov 03 17:44:50.871647 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.871633    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfmqk\" (UniqueName: \"kubernetes.io/projected/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-kube-api-access-rfmqk\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:44:50.871647 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.871639    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzjx9\" (UniqueName: \"kubernetes.io/projected/d59f146b-82ee-4f26-854a-cba274a89600-kube-api-access-bzjx9\") pod \"aws-ebs-csi-driver-node-v6bks\" (UID: \"d59f146b-82ee-4f26-854a-cba274a89600\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6bks"
Nov 03 17:44:50.871752 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.871625    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpp49\" (UniqueName: \"kubernetes.io/projected/a831ae97-8f46-4b85-bfdb-59042b7b8b0c-kube-api-access-mpp49\") pod \"node-resolver-9j547\" (UID: \"a831ae97-8f46-4b85-bfdb-59042b7b8b0c\") " pod="openshift-dns/node-resolver-9j547"
Nov 03 17:44:50.974984 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.974929    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:44:50.996267 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:50.996235    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-n9qsq"
Nov 03 17:44:51.022669 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:51.022646    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bsgc5"
Nov 03 17:44:51.025764 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:51.025747    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jbbwp"
Nov 03 17:44:51.031222 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:51.031175    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a90b512115a10076b69387005e99589.slice/crio-abb3d20d17c9e1057141552b05e0c65826502f65fcb07c44d2c74c00e481b7f1 WatchSource:0}: Error finding container abb3d20d17c9e1057141552b05e0c65826502f65fcb07c44d2c74c00e481b7f1: Status 404 returned error can't find the container with id abb3d20d17c9e1057141552b05e0c65826502f65fcb07c44d2c74c00e481b7f1
Nov 03 17:44:51.031531 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:51.031510    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dbd524262c9b524921d913509ef36fa.slice/crio-735b63977e239842f058114ff665b7db5d04b38dd6ab509212b94544eaee250d WatchSource:0}: Error finding container 735b63977e239842f058114ff665b7db5d04b38dd6ab509212b94544eaee250d: Status 404 returned error can't find the container with id 735b63977e239842f058114ff665b7db5d04b38dd6ab509212b94544eaee250d
Nov 03 17:44:51.032263 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:51.032240    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod778383aa_b3db_45a6_8bec_9891f7818e94.slice/crio-17f5aed41c2adc3c7122f2af59303d8dd656fbdcfaba6b7b85cdd74ede8e5804 WatchSource:0}: Error finding container 17f5aed41c2adc3c7122f2af59303d8dd656fbdcfaba6b7b85cdd74ede8e5804: Status 404 returned error can't find the container with id 17f5aed41c2adc3c7122f2af59303d8dd656fbdcfaba6b7b85cdd74ede8e5804
Nov 03 17:44:51.036574 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:51.036554    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 17:44:51.037440 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:51.037418    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb993b313_db8b_42f5_ad99_2d2ab5bb49e0.slice/crio-e30ab61f1fcccd658bc913224880d4de5f884763621080cf78cb3880b3cef69f WatchSource:0}: Error finding container e30ab61f1fcccd658bc913224880d4de5f884763621080cf78cb3880b3cef69f: Status 404 returned error can't find the container with id e30ab61f1fcccd658bc913224880d4de5f884763621080cf78cb3880b3cef69f
Nov 03 17:44:51.038867 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:51.038396    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a8c802b_6b86_41cd_ae81_bb3ed0b6ba4e.slice/crio-e6269ad31b395b408406d12e3bfbab8b2317923cdf821b7c815c69caac58878a WatchSource:0}: Error finding container e6269ad31b395b408406d12e3bfbab8b2317923cdf821b7c815c69caac58878a: Status 404 returned error can't find the container with id e6269ad31b395b408406d12e3bfbab8b2317923cdf821b7c815c69caac58878a
Nov 03 17:44:51.041694 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:51.041681    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:44:51.047259 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:51.047241    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6bks"
Nov 03 17:44:51.049813 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:51.049797    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc29d200a_c917_46e5_9cf5_21f4d952e470.slice/crio-e9ad1fb5061bef579da494a3748707e9bc6214af69ee74945e6254c3ce22365f WatchSource:0}: Error finding container e9ad1fb5061bef579da494a3748707e9bc6214af69ee74945e6254c3ce22365f: Status 404 returned error can't find the container with id e9ad1fb5061bef579da494a3748707e9bc6214af69ee74945e6254c3ce22365f
Nov 03 17:44:51.054468 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:51.054451    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd59f146b_82ee_4f26_854a_cba274a89600.slice/crio-8549638c99969e115d70c745d0c81574071f2c1ac772f71c1dff07aaa52634aa WatchSource:0}: Error finding container 8549638c99969e115d70c745d0c81574071f2c1ac772f71c1dff07aaa52634aa: Status 404 returned error can't find the container with id 8549638c99969e115d70c745d0c81574071f2c1ac772f71c1dff07aaa52634aa
Nov 03 17:44:51.060248 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:51.060232    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9j547"
Nov 03 17:44:51.068261 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:51.068244    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda831ae97_8f46_4b85_bfdb_59042b7b8b0c.slice/crio-1794a07f998f56a2861ae8bc703815fb34354f4e54a42e590f6a7535ced033d9 WatchSource:0}: Error finding container 1794a07f998f56a2861ae8bc703815fb34354f4e54a42e590f6a7535ced033d9: Status 404 returned error can't find the container with id 1794a07f998f56a2861ae8bc703815fb34354f4e54a42e590f6a7535ced033d9
Nov 03 17:44:51.077897 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:51.077879    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-x8f2p"
Nov 03 17:44:51.083335 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:51.083316    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dv4zl"
Nov 03 17:44:51.086030 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:51.086012    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79212f04_d10c_43ee_9dd5_41b4fd1917b2.slice/crio-00bbb7d679385391ad209f99b21a55fceb70b6ed883a692a74de60e11e9024d6 WatchSource:0}: Error finding container 00bbb7d679385391ad209f99b21a55fceb70b6ed883a692a74de60e11e9024d6: Status 404 returned error can't find the container with id 00bbb7d679385391ad209f99b21a55fceb70b6ed883a692a74de60e11e9024d6
Nov 03 17:44:51.090779 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:51.090756    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb91c00dc_c14f_4b3a_b268_1eb6dcf9ad4b.slice/crio-3ad8db60b8760552cac5c6731018d7b4d80e8e8faf4944fcd6e0ec4fc4fdad1a WatchSource:0}: Error finding container 3ad8db60b8760552cac5c6731018d7b4d80e8e8faf4944fcd6e0ec4fc4fdad1a: Status 404 returned error can't find the container with id 3ad8db60b8760552cac5c6731018d7b4d80e8e8faf4944fcd6e0ec4fc4fdad1a
Nov 03 17:44:51.266731 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:51.266663    2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:44:51.314651 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:51.314623    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cf65c17-1c79-4cd5-9ef1-dceb467878b0-metrics-certs\") pod \"network-metrics-daemon-z264t\" (UID: \"9cf65c17-1c79-4cd5-9ef1-dceb467878b0\") " pod="openshift-multus/network-metrics-daemon-z264t"
Nov 03 17:44:51.314802 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:51.314658    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30-original-pull-secret\") pod \"global-pull-secret-syncer-4xjw2\" (UID: \"f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30\") " pod="kube-system/global-pull-secret-syncer-4xjw2"
Nov 03 17:44:51.314802 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:51.314746    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 17:44:51.314802 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:51.314752    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:44:51.314802 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:51.314790    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30-original-pull-secret podName:f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30 nodeName:}" failed. No retries permitted until 2026-11-03 17:44:52.314779779 +0000 UTC m=+3.290427499 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30-original-pull-secret") pod "global-pull-secret-syncer-4xjw2" (UID: "f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30") : object "kube-system"/"original-pull-secret" not registered
Nov 03 17:44:51.314918 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:51.314811    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cf65c17-1c79-4cd5-9ef1-dceb467878b0-metrics-certs podName:9cf65c17-1c79-4cd5-9ef1-dceb467878b0 nodeName:}" failed. No retries permitted until 2026-11-03 17:44:52.3147969 +0000 UTC m=+3.290444620 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9cf65c17-1c79-4cd5-9ef1-dceb467878b0-metrics-certs") pod "network-metrics-daemon-z264t" (UID: "9cf65c17-1c79-4cd5-9ef1-dceb467878b0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:44:51.414943 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:51.414905    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v89s6\" (UniqueName: \"kubernetes.io/projected/43c9da28-334d-4bb3-a6bc-e3da2d561633-kube-api-access-v89s6\") pod \"network-check-target-2t57w\" (UID: \"43c9da28-334d-4bb3-a6bc-e3da2d561633\") " pod="openshift-network-diagnostics/network-check-target-2t57w"
Nov 03 17:44:51.415549 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:51.415021    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 17:44:51.415549 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:51.415034    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 17:44:51.415549 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:51.415043    2576 projected.go:196] Error preparing data for projected volume kube-api-access-v89s6 for pod openshift-network-diagnostics/network-check-target-2t57w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:44:51.415549 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:51.415087    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43c9da28-334d-4bb3-a6bc-e3da2d561633-kube-api-access-v89s6 podName:43c9da28-334d-4bb3-a6bc-e3da2d561633 nodeName:}" failed. No retries permitted until 2026-11-03 17:44:52.41507473 +0000 UTC m=+3.390722452 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-v89s6" (UniqueName: "kubernetes.io/projected/43c9da28-334d-4bb3-a6bc-e3da2d561633-kube-api-access-v89s6") pod "network-check-target-2t57w" (UID: "43c9da28-334d-4bb3-a6bc-e3da2d561633") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:44:51.560007 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:51.559911    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dv4zl" event={"ID":"b91c00dc-c14f-4b3a-b268-1eb6dcf9ad4b","Type":"ContainerStarted","Data":"3ad8db60b8760552cac5c6731018d7b4d80e8e8faf4944fcd6e0ec4fc4fdad1a"}
Nov 03 17:44:51.561057 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:51.561032    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9j547" event={"ID":"a831ae97-8f46-4b85-bfdb-59042b7b8b0c","Type":"ContainerStarted","Data":"1794a07f998f56a2861ae8bc703815fb34354f4e54a42e590f6a7535ced033d9"}
Nov 03 17:44:51.562034 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:51.562007    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6bks" event={"ID":"d59f146b-82ee-4f26-854a-cba274a89600","Type":"ContainerStarted","Data":"8549638c99969e115d70c745d0c81574071f2c1ac772f71c1dff07aaa52634aa"}
Nov 03 17:44:51.562935 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:51.562908    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bsgc5" event={"ID":"b993b313-db8b-42f5-ad99-2d2ab5bb49e0","Type":"ContainerStarted","Data":"e30ab61f1fcccd658bc913224880d4de5f884763621080cf78cb3880b3cef69f"}
Nov 03 17:44:51.563870 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:51.563850    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-204.us-west-2.compute.internal" event={"ID":"8dbd524262c9b524921d913509ef36fa","Type":"ContainerStarted","Data":"735b63977e239842f058114ff665b7db5d04b38dd6ab509212b94544eaee250d"}
Nov 03 17:44:51.564785 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:51.564767    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-204.us-west-2.compute.internal" event={"ID":"8a90b512115a10076b69387005e99589","Type":"ContainerStarted","Data":"abb3d20d17c9e1057141552b05e0c65826502f65fcb07c44d2c74c00e481b7f1"}
Nov 03 17:44:51.565647 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:51.565627    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-x8f2p" event={"ID":"79212f04-d10c-43ee-9dd5-41b4fd1917b2","Type":"ContainerStarted","Data":"00bbb7d679385391ad209f99b21a55fceb70b6ed883a692a74de60e11e9024d6"}
Nov 03 17:44:51.566631 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:51.566596    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-97p65" event={"ID":"c29d200a-c917-46e5-9cf5-21f4d952e470","Type":"ContainerStarted","Data":"e9ad1fb5061bef579da494a3748707e9bc6214af69ee74945e6254c3ce22365f"}
Nov 03 17:44:51.567577 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:51.567549    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jbbwp" event={"ID":"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e","Type":"ContainerStarted","Data":"e6269ad31b395b408406d12e3bfbab8b2317923cdf821b7c815c69caac58878a"}
Nov 03 17:44:51.568458 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:51.568440    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-n9qsq" event={"ID":"778383aa-b3db-45a6-8bec-9891f7818e94","Type":"ContainerStarted","Data":"17f5aed41c2adc3c7122f2af59303d8dd656fbdcfaba6b7b85cdd74ede8e5804"}
Nov 03 17:44:51.781543 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:51.781491    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 17:39:50 +0000 UTC" deadline="2028-04-11 13:11:29.879129552 +0000 UTC"
Nov 03 17:44:51.781543 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:51.781535    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12595h26m38.097597064s"
Nov 03 17:44:51.806790 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:51.806762    2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:44:51.832956 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:44:51.832924    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce15fea2_6b11_4b2b_8abb_7ec29840c78f.slice/crio-095dbbdb7fcecb08dfd135a2b637518d1323c71744ae72c22ffbca0837808914 WatchSource:0}: Error finding container 095dbbdb7fcecb08dfd135a2b637518d1323c71744ae72c22ffbca0837808914: Status 404 returned error can't find the container with id 095dbbdb7fcecb08dfd135a2b637518d1323c71744ae72c22ffbca0837808914
Nov 03 17:44:51.833535 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:51.833518    2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:44:52.320776 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:52.320695    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cf65c17-1c79-4cd5-9ef1-dceb467878b0-metrics-certs\") pod \"network-metrics-daemon-z264t\" (UID: \"9cf65c17-1c79-4cd5-9ef1-dceb467878b0\") " pod="openshift-multus/network-metrics-daemon-z264t"
Nov 03 17:44:52.320776 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:52.320757    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30-original-pull-secret\") pod \"global-pull-secret-syncer-4xjw2\" (UID: \"f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30\") " pod="kube-system/global-pull-secret-syncer-4xjw2"
Nov 03 17:44:52.321001 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:52.320896    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 17:44:52.321001 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:52.320943    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30-original-pull-secret podName:f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30 nodeName:}" failed. No retries permitted until 2026-11-03 17:44:54.320931288 +0000 UTC m=+5.296579010 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30-original-pull-secret") pod "global-pull-secret-syncer-4xjw2" (UID: "f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30") : object "kube-system"/"original-pull-secret" not registered
Nov 03 17:44:52.321001 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:52.320988    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:44:52.321156 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:52.321012    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cf65c17-1c79-4cd5-9ef1-dceb467878b0-metrics-certs podName:9cf65c17-1c79-4cd5-9ef1-dceb467878b0 nodeName:}" failed. No retries permitted until 2026-11-03 17:44:54.321004234 +0000 UTC m=+5.296651956 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9cf65c17-1c79-4cd5-9ef1-dceb467878b0-metrics-certs") pod "network-metrics-daemon-z264t" (UID: "9cf65c17-1c79-4cd5-9ef1-dceb467878b0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:44:52.421099 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:52.421063    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v89s6\" (UniqueName: \"kubernetes.io/projected/43c9da28-334d-4bb3-a6bc-e3da2d561633-kube-api-access-v89s6\") pod \"network-check-target-2t57w\" (UID: \"43c9da28-334d-4bb3-a6bc-e3da2d561633\") " pod="openshift-network-diagnostics/network-check-target-2t57w"
Nov 03 17:44:52.421658 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:52.421190    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 17:44:52.421658 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:52.421203    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 17:44:52.421658 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:52.421214    2576 projected.go:196] Error preparing data for projected volume kube-api-access-v89s6 for pod openshift-network-diagnostics/network-check-target-2t57w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:44:52.421658 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:52.421260    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43c9da28-334d-4bb3-a6bc-e3da2d561633-kube-api-access-v89s6 podName:43c9da28-334d-4bb3-a6bc-e3da2d561633 nodeName:}" failed. No retries permitted until 2026-11-03 17:44:54.421243184 +0000 UTC m=+5.396890920 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-v89s6" (UniqueName: "kubernetes.io/projected/43c9da28-334d-4bb3-a6bc-e3da2d561633-kube-api-access-v89s6") pod "network-check-target-2t57w" (UID: "43c9da28-334d-4bb3-a6bc-e3da2d561633") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:44:52.555553 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:52.555517    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xjw2"
Nov 03 17:44:52.555742 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:52.555667    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xjw2" podUID="f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30"
Nov 03 17:44:52.555803 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:52.555757    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z264t"
Nov 03 17:44:52.555889 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:52.555841    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z264t" podUID="9cf65c17-1c79-4cd5-9ef1-dceb467878b0"
Nov 03 17:44:52.555942 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:52.555893    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2t57w"
Nov 03 17:44:52.556044 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:52.555957    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2t57w" podUID="43c9da28-334d-4bb3-a6bc-e3da2d561633"
Nov 03 17:44:52.589318 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:52.589233    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fjrg4" event={"ID":"ce15fea2-6b11-4b2b-8abb-7ec29840c78f","Type":"ContainerStarted","Data":"095dbbdb7fcecb08dfd135a2b637518d1323c71744ae72c22ffbca0837808914"}
Nov 03 17:44:52.782506 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:52.782423    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 17:39:50 +0000 UTC" deadline="2028-08-07 09:14:31.4860249 +0000 UTC"
Nov 03 17:44:52.782506 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:52.782463    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15423h29m38.703567012s"
Nov 03 17:44:54.340143 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:54.340105    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cf65c17-1c79-4cd5-9ef1-dceb467878b0-metrics-certs\") pod \"network-metrics-daemon-z264t\" (UID: \"9cf65c17-1c79-4cd5-9ef1-dceb467878b0\") " pod="openshift-multus/network-metrics-daemon-z264t"
Nov 03 17:44:54.340629 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:54.340168    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30-original-pull-secret\") pod \"global-pull-secret-syncer-4xjw2\" (UID: \"f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30\") " pod="kube-system/global-pull-secret-syncer-4xjw2"
Nov 03 17:44:54.340629 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:54.340330    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 17:44:54.340629 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:54.340391    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30-original-pull-secret podName:f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30 nodeName:}" failed. No retries permitted until 2026-11-03 17:44:58.34037555 +0000 UTC m=+9.316023286 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30-original-pull-secret") pod "global-pull-secret-syncer-4xjw2" (UID: "f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30") : object "kube-system"/"original-pull-secret" not registered
Nov 03 17:44:54.340629 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:54.340460    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:44:54.340629 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:54.340493    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cf65c17-1c79-4cd5-9ef1-dceb467878b0-metrics-certs podName:9cf65c17-1c79-4cd5-9ef1-dceb467878b0 nodeName:}" failed. No retries permitted until 2026-11-03 17:44:58.340481629 +0000 UTC m=+9.316129365 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9cf65c17-1c79-4cd5-9ef1-dceb467878b0-metrics-certs") pod "network-metrics-daemon-z264t" (UID: "9cf65c17-1c79-4cd5-9ef1-dceb467878b0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:44:54.440927 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:54.440891    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v89s6\" (UniqueName: \"kubernetes.io/projected/43c9da28-334d-4bb3-a6bc-e3da2d561633-kube-api-access-v89s6\") pod \"network-check-target-2t57w\" (UID: \"43c9da28-334d-4bb3-a6bc-e3da2d561633\") " pod="openshift-network-diagnostics/network-check-target-2t57w"
Nov 03 17:44:54.441102 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:54.441069    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 17:44:54.441102 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:54.441087    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 17:44:54.441275 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:54.441163    2576 projected.go:196] Error preparing data for projected volume kube-api-access-v89s6 for pod openshift-network-diagnostics/network-check-target-2t57w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:44:54.441275 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:54.441224    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43c9da28-334d-4bb3-a6bc-e3da2d561633-kube-api-access-v89s6 podName:43c9da28-334d-4bb3-a6bc-e3da2d561633 nodeName:}" failed. No retries permitted until 2026-11-03 17:44:58.441205767 +0000 UTC m=+9.416853495 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-v89s6" (UniqueName: "kubernetes.io/projected/43c9da28-334d-4bb3-a6bc-e3da2d561633-kube-api-access-v89s6") pod "network-check-target-2t57w" (UID: "43c9da28-334d-4bb3-a6bc-e3da2d561633") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:44:54.556510 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:54.556472    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xjw2"
Nov 03 17:44:54.556725 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:54.556634    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xjw2" podUID="f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30"
Nov 03 17:44:54.556789 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:54.556739    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z264t"
Nov 03 17:44:54.556923 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:54.556843    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z264t" podUID="9cf65c17-1c79-4cd5-9ef1-dceb467878b0"
Nov 03 17:44:54.556923 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:54.556903    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2t57w"
Nov 03 17:44:54.557046 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:54.556967    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2t57w" podUID="43c9da28-334d-4bb3-a6bc-e3da2d561633"
Nov 03 17:44:55.622852 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:55.622770    2576 generic.go:356] "Generic (PLEG): container finished" podID="b993b313-db8b-42f5-ad99-2d2ab5bb49e0" containerID="8c33a28ab106380469ba6734489f9bb1b0116dc8b9a23d5a1fab50d38f6a2ee2" exitCode=0
Nov 03 17:44:55.622852 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:55.622832    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bsgc5" event={"ID":"b993b313-db8b-42f5-ad99-2d2ab5bb49e0","Type":"ContainerDied","Data":"8c33a28ab106380469ba6734489f9bb1b0116dc8b9a23d5a1fab50d38f6a2ee2"}
Nov 03 17:44:56.556090 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:56.555851    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xjw2"
Nov 03 17:44:56.556217 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:56.556200    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xjw2" podUID="f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30"
Nov 03 17:44:56.556279 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:56.556267    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z264t"
Nov 03 17:44:56.556379 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:56.556352    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z264t" podUID="9cf65c17-1c79-4cd5-9ef1-dceb467878b0"
Nov 03 17:44:56.556419 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:56.556397    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2t57w"
Nov 03 17:44:56.556481 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:56.556455    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2t57w" podUID="43c9da28-334d-4bb3-a6bc-e3da2d561633"
Nov 03 17:44:56.634147 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:56.633461    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dv4zl" event={"ID":"b91c00dc-c14f-4b3a-b268-1eb6dcf9ad4b","Type":"ContainerStarted","Data":"a449a6a8fd584cbd4aa945dc82fe45a8d6f5bb93985272163ca38e4504f67b54"}
Nov 03 17:44:56.637179 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:56.637115    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-204.us-west-2.compute.internal" event={"ID":"8dbd524262c9b524921d913509ef36fa","Type":"ContainerStarted","Data":"b07f87172a975d93912a71a3d2ab236f4ad0063091498841cad16ef0f24c6b2b"}
Nov 03 17:44:56.642806 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:56.642042    2576 generic.go:356] "Generic (PLEG): container finished" podID="8a90b512115a10076b69387005e99589" containerID="610f607107d75973ef19289a402337ab681768975df03a11d39328fb87704041" exitCode=0
Nov 03 17:44:56.642806 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:56.642088    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-204.us-west-2.compute.internal" event={"ID":"8a90b512115a10076b69387005e99589","Type":"ContainerDied","Data":"610f607107d75973ef19289a402337ab681768975df03a11d39328fb87704041"}
Nov 03 17:44:56.871651 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:56.870857    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dv4zl" podStartSLOduration=3.213015136 podStartE2EDuration="7.870836063s" totalImagesPullingTime="4.657820927s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:44:56.752800135 +0000 UTC m=+7.728447877" watchObservedRunningTime="2026-11-03 17:44:56.870836063 +0000 UTC m=+7.846483811"
Nov 03 17:44:57.648620 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:57.648574    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-204.us-west-2.compute.internal" event={"ID":"8a90b512115a10076b69387005e99589","Type":"ContainerStarted","Data":"2636ed70d99c558451ee0631f4094289029280c136380ade1436ead2b49a1d49"}
Nov 03 17:44:57.652153 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:57.651589    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-x8f2p" event={"ID":"79212f04-d10c-43ee-9dd5-41b4fd1917b2","Type":"ContainerStarted","Data":"e919d96c0e70d8f84e821d5d4bb73750370f85cfed3726a57e825a266dff4e72"}
Nov 03 17:44:57.767357 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:57.767296    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-204.us-west-2.compute.internal" podStartSLOduration=8.767276121 podStartE2EDuration="8.767276121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:44:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:44:56.991189008 +0000 UTC m=+7.966836754" watchObservedRunningTime="2026-11-03 17:44:57.767276121 +0000 UTC m=+8.742923867"
Nov 03 17:44:57.767812 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:57.767765    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-204.us-west-2.compute.internal" podStartSLOduration=8.767751567 podStartE2EDuration="8.767751567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:44:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:44:57.766854472 +0000 UTC m=+8.742502274" watchObservedRunningTime="2026-11-03 17:44:57.767751567 +0000 UTC m=+8.743399312"
Nov 03 17:44:58.379663 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:58.379313    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cf65c17-1c79-4cd5-9ef1-dceb467878b0-metrics-certs\") pod \"network-metrics-daemon-z264t\" (UID: \"9cf65c17-1c79-4cd5-9ef1-dceb467878b0\") " pod="openshift-multus/network-metrics-daemon-z264t"
Nov 03 17:44:58.379663 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:58.379356    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30-original-pull-secret\") pod \"global-pull-secret-syncer-4xjw2\" (UID: \"f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30\") " pod="kube-system/global-pull-secret-syncer-4xjw2"
Nov 03 17:44:58.379663 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:58.379478    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 17:44:58.379663 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:58.379523    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30-original-pull-secret podName:f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:06.379511851 +0000 UTC m=+17.355159585 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30-original-pull-secret") pod "global-pull-secret-syncer-4xjw2" (UID: "f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30") : object "kube-system"/"original-pull-secret" not registered
Nov 03 17:44:58.379663 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:58.379563    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:44:58.379663 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:58.379582    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cf65c17-1c79-4cd5-9ef1-dceb467878b0-metrics-certs podName:9cf65c17-1c79-4cd5-9ef1-dceb467878b0 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:06.379575443 +0000 UTC m=+17.355223168 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9cf65c17-1c79-4cd5-9ef1-dceb467878b0-metrics-certs") pod "network-metrics-daemon-z264t" (UID: "9cf65c17-1c79-4cd5-9ef1-dceb467878b0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:44:58.481081 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:58.480718    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v89s6\" (UniqueName: \"kubernetes.io/projected/43c9da28-334d-4bb3-a6bc-e3da2d561633-kube-api-access-v89s6\") pod \"network-check-target-2t57w\" (UID: \"43c9da28-334d-4bb3-a6bc-e3da2d561633\") " pod="openshift-network-diagnostics/network-check-target-2t57w"
Nov 03 17:44:58.481081 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:58.480894    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 17:44:58.481081 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:58.480917    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 17:44:58.481081 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:58.480931    2576 projected.go:196] Error preparing data for projected volume kube-api-access-v89s6 for pod openshift-network-diagnostics/network-check-target-2t57w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:44:58.481081 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:58.481052    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43c9da28-334d-4bb3-a6bc-e3da2d561633-kube-api-access-v89s6 podName:43c9da28-334d-4bb3-a6bc-e3da2d561633 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:06.481033625 +0000 UTC m=+17.456681351 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-v89s6" (UniqueName: "kubernetes.io/projected/43c9da28-334d-4bb3-a6bc-e3da2d561633-kube-api-access-v89s6") pod "network-check-target-2t57w" (UID: "43c9da28-334d-4bb3-a6bc-e3da2d561633") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:44:58.555775 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:58.555744    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2t57w"
Nov 03 17:44:58.555953 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:58.555863    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2t57w" podUID="43c9da28-334d-4bb3-a6bc-e3da2d561633"
Nov 03 17:44:58.555953 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:58.555931    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xjw2"
Nov 03 17:44:58.556068 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:58.556004    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xjw2" podUID="f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30"
Nov 03 17:44:58.556068 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:58.556059    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z264t"
Nov 03 17:44:58.556171 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:44:58.556138    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z264t" podUID="9cf65c17-1c79-4cd5-9ef1-dceb467878b0"
Nov 03 17:44:58.657001 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:58.656908    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6bks" event={"ID":"d59f146b-82ee-4f26-854a-cba274a89600","Type":"ContainerStarted","Data":"fcea85ca56d7734b3d9c68e2661868f4567307c571e4567d5d472031888f5147"}
Nov 03 17:44:59.666683 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:59.666223    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jbbwp" event={"ID":"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e","Type":"ContainerStarted","Data":"2b83b0bcb7b457d72d928b937d6f7d05877d5674fe362cc7f701cb5e179eadbf"}
Nov 03 17:44:59.794043 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:59.793991    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-x8f2p" podStartSLOduration=4.812066226 podStartE2EDuration="10.793974585s" totalImagesPullingTime="5.981908359s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:44:57.898181385 +0000 UTC m=+8.873829132" watchObservedRunningTime="2026-11-03 17:44:59.793974585 +0000 UTC m=+10.769622330"
Nov 03 17:44:59.820169 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:59.820144    2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 17:44:59.944318 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:59.944288    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dv4zl"
Nov 03 17:44:59.945485 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:44:59.945465    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dv4zl"
Nov 03 17:45:00.067831 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:00.067779    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-jbbwp" podStartSLOduration=2.6343526390000003 podStartE2EDuration="11.067762046s" totalImagesPullingTime="8.433409407s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:44:59.794342111 +0000 UTC m=+10.769989855" watchObservedRunningTime="2026-11-03 17:45:00.067762046 +0000 UTC m=+11.043409790"
Nov 03 17:45:00.463710 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:00.463546    2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T17:44:59.820165873Z","UUID":"590b366c-5e98-4e3e-95f7-4a135831b6ac","Handler":null,"Name":"","Endpoint":""}
Nov 03 17:45:00.465159 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:00.465142    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 17:45:00.465278 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:00.465170    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 17:45:00.556573 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:00.556500    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2t57w"
Nov 03 17:45:00.556760 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:00.556571    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z264t"
Nov 03 17:45:00.556760 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:00.556711    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xjw2"
Nov 03 17:45:00.556760 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:00.556729    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2t57w" podUID="43c9da28-334d-4bb3-a6bc-e3da2d561633"
Nov 03 17:45:00.556935 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:00.556814    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xjw2" podUID="f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30"
Nov 03 17:45:00.556935 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:00.556919    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z264t" podUID="9cf65c17-1c79-4cd5-9ef1-dceb467878b0"
Nov 03 17:45:00.671245 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:00.671210    2576 generic.go:356] "Generic (PLEG): container finished" podID="b993b313-db8b-42f5-ad99-2d2ab5bb49e0" containerID="1014f827db955aeaea5b0c6ff4029b5aa0c33a5e0d581a5770fb435e2686be13" exitCode=0
Nov 03 17:45:00.671735 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:00.671301    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bsgc5" event={"ID":"b993b313-db8b-42f5-ad99-2d2ab5bb49e0","Type":"ContainerDied","Data":"1014f827db955aeaea5b0c6ff4029b5aa0c33a5e0d581a5770fb435e2686be13"}
Nov 03 17:45:00.673817 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:00.673775    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9j547" event={"ID":"a831ae97-8f46-4b85-bfdb-59042b7b8b0c","Type":"ContainerStarted","Data":"d1649ec8246b02c4aa033656147e6d6daa908c39f93e9dbf05bb069a00a0674f"}
Nov 03 17:45:00.676873 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:00.676814    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6bks" event={"ID":"d59f146b-82ee-4f26-854a-cba274a89600","Type":"ContainerStarted","Data":"7cd21cd072c6235aeb2cd1f5870a42bf3193c4b91447309797b724b5df7fe24d"}
Nov 03 17:45:00.915191 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:00.914359    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-9j547" podStartSLOduration=3.414428311 podStartE2EDuration="11.914338486s" totalImagesPullingTime="8.499910175s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:00.914045974 +0000 UTC m=+11.889693718" watchObservedRunningTime="2026-11-03 17:45:00.914338486 +0000 UTC m=+11.889986256"
Nov 03 17:45:00.928275 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:00.928185    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dv4zl"
Nov 03 17:45:00.928965 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:00.928938    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dv4zl"
Nov 03 17:45:01.681111 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:01.681069    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-n9qsq" event={"ID":"778383aa-b3db-45a6-8bec-9891f7818e94","Type":"ContainerStarted","Data":"933c9361d7838482b4da1e98e2ba814e7381d021fa19144316327c379b5c50c6"}
Nov 03 17:45:01.684154 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:01.684102    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6bks" event={"ID":"d59f146b-82ee-4f26-854a-cba274a89600","Type":"ContainerStarted","Data":"a79df91775dbedb46dfc1d50119698c19a9f63ad204222f68b820bd6b9422b4b"}
Nov 03 17:45:01.800412 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:01.800193    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-n9qsq" podStartSLOduration=3.869015771 podStartE2EDuration="12.800177357s" totalImagesPullingTime="8.931161586s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:01.799871792 +0000 UTC m=+12.775519541" watchObservedRunningTime="2026-11-03 17:45:01.800177357 +0000 UTC m=+12.775825101"
Nov 03 17:45:01.922292 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:01.922226    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6bks" podStartSLOduration=3.093018074 podStartE2EDuration="12.92220328s" totalImagesPullingTime="9.829185206s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:44:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:01.921070243 +0000 UTC m=+12.896717987" watchObservedRunningTime="2026-11-03 17:45:01.92220328 +0000 UTC m=+12.897851037"
Nov 03 17:45:02.556119 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:02.556085    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z264t"
Nov 03 17:45:02.556119 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:02.556126    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2t57w"
Nov 03 17:45:02.556456 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:02.556209    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xjw2"
Nov 03 17:45:02.556456 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:02.556214    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z264t" podUID="9cf65c17-1c79-4cd5-9ef1-dceb467878b0"
Nov 03 17:45:02.556456 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:02.556328    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xjw2" podUID="f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30"
Nov 03 17:45:02.556456 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:02.556414    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2t57w" podUID="43c9da28-334d-4bb3-a6bc-e3da2d561633"
Nov 03 17:45:02.688140 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:02.688050    2576 generic.go:356] "Generic (PLEG): container finished" podID="b993b313-db8b-42f5-ad99-2d2ab5bb49e0" containerID="84d85ef08a77cf63d35185a436052f4646f0d83e500ece74262104ed17661fba" exitCode=0
Nov 03 17:45:02.688602 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:02.688138    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bsgc5" event={"ID":"b993b313-db8b-42f5-ad99-2d2ab5bb49e0","Type":"ContainerDied","Data":"84d85ef08a77cf63d35185a436052f4646f0d83e500ece74262104ed17661fba"}
Nov 03 17:45:02.689826 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:02.689791    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fjrg4" event={"ID":"ce15fea2-6b11-4b2b-8abb-7ec29840c78f","Type":"ContainerStarted","Data":"04b06e731b4be86b52eb10b1d889902a551332130ab123c1d2d232099459b411"}
Nov 03 17:45:02.934052 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:02.934000    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fjrg4" podStartSLOduration=3.401783682 podStartE2EDuration="13.933988293s" totalImagesPullingTime="10.532204611s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:02.93370708 +0000 UTC m=+13.909354821" watchObservedRunningTime="2026-11-03 17:45:02.933988293 +0000 UTC m=+13.909636038"
Nov 03 17:45:04.556400 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:04.556359    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2t57w"
Nov 03 17:45:04.557142 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:04.556376    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xjw2"
Nov 03 17:45:04.557142 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:04.556502    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2t57w" podUID="43c9da28-334d-4bb3-a6bc-e3da2d561633"
Nov 03 17:45:04.557142 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:04.556380    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z264t"
Nov 03 17:45:04.557142 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:04.556569    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xjw2" podUID="f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30"
Nov 03 17:45:04.557142 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:04.556632    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z264t" podUID="9cf65c17-1c79-4cd5-9ef1-dceb467878b0"
Nov 03 17:45:04.696877 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:04.696842    2576 generic.go:356] "Generic (PLEG): container finished" podID="b993b313-db8b-42f5-ad99-2d2ab5bb49e0" containerID="664ccb3c5f6b0ffebf87cfce1d29ebe7d6f202ab00bc14b157430c00f377fd75" exitCode=0
Nov 03 17:45:04.697038 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:04.696904    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bsgc5" event={"ID":"b993b313-db8b-42f5-ad99-2d2ab5bb49e0","Type":"ContainerDied","Data":"664ccb3c5f6b0ffebf87cfce1d29ebe7d6f202ab00bc14b157430c00f377fd75"}
Nov 03 17:45:06.448510 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:06.448470    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cf65c17-1c79-4cd5-9ef1-dceb467878b0-metrics-certs\") pod \"network-metrics-daemon-z264t\" (UID: \"9cf65c17-1c79-4cd5-9ef1-dceb467878b0\") " pod="openshift-multus/network-metrics-daemon-z264t"
Nov 03 17:45:06.449016 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:06.448537    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30-original-pull-secret\") pod \"global-pull-secret-syncer-4xjw2\" (UID: \"f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30\") " pod="kube-system/global-pull-secret-syncer-4xjw2"
Nov 03 17:45:06.449016 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:06.448598    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:45:06.449016 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:06.448678    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 17:45:06.449016 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:06.448696    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cf65c17-1c79-4cd5-9ef1-dceb467878b0-metrics-certs podName:9cf65c17-1c79-4cd5-9ef1-dceb467878b0 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:22.448677418 +0000 UTC m=+33.424325163 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9cf65c17-1c79-4cd5-9ef1-dceb467878b0-metrics-certs") pod "network-metrics-daemon-z264t" (UID: "9cf65c17-1c79-4cd5-9ef1-dceb467878b0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:45:06.449016 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:06.448738    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30-original-pull-secret podName:f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:22.448719673 +0000 UTC m=+33.424367410 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30-original-pull-secret") pod "global-pull-secret-syncer-4xjw2" (UID: "f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30") : object "kube-system"/"original-pull-secret" not registered
Nov 03 17:45:06.549785 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:06.549519    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v89s6\" (UniqueName: \"kubernetes.io/projected/43c9da28-334d-4bb3-a6bc-e3da2d561633-kube-api-access-v89s6\") pod \"network-check-target-2t57w\" (UID: \"43c9da28-334d-4bb3-a6bc-e3da2d561633\") " pod="openshift-network-diagnostics/network-check-target-2t57w"
Nov 03 17:45:06.549880 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:06.549710    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 17:45:06.549929 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:06.549884    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 17:45:06.549929 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:06.549902    2576 projected.go:196] Error preparing data for projected volume kube-api-access-v89s6 for pod openshift-network-diagnostics/network-check-target-2t57w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:45:06.550001 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:06.549967    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43c9da28-334d-4bb3-a6bc-e3da2d561633-kube-api-access-v89s6 podName:43c9da28-334d-4bb3-a6bc-e3da2d561633 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:22.549948066 +0000 UTC m=+33.525595805 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-v89s6" (UniqueName: "kubernetes.io/projected/43c9da28-334d-4bb3-a6bc-e3da2d561633-kube-api-access-v89s6") pod "network-check-target-2t57w" (UID: "43c9da28-334d-4bb3-a6bc-e3da2d561633") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:45:06.555645 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:06.555602    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xjw2"
Nov 03 17:45:06.555746 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:06.555697    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2t57w"
Nov 03 17:45:06.555823 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:06.555803    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z264t"
Nov 03 17:45:06.555882 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:06.555825    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xjw2" podUID="f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30"
Nov 03 17:45:06.555973 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:06.555953    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z264t" podUID="9cf65c17-1c79-4cd5-9ef1-dceb467878b0"
Nov 03 17:45:06.556060 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:06.556037    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2t57w" podUID="43c9da28-334d-4bb3-a6bc-e3da2d561633"
Nov 03 17:45:06.703348 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:06.703327    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-97p65_c29d200a-c917-46e5-9cf5-21f4d952e470/ovn-acl-logging/0.log"
Nov 03 17:45:06.703764 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:06.703730    2576 generic.go:356] "Generic (PLEG): container finished" podID="c29d200a-c917-46e5-9cf5-21f4d952e470" containerID="a068e219a50a01990dd55ec22bf5723658419d68bca038fbea25f88e779d0e78" exitCode=1
Nov 03 17:45:06.703843 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:06.703774    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-97p65" event={"ID":"c29d200a-c917-46e5-9cf5-21f4d952e470","Type":"ContainerStarted","Data":"12d026efd28bf43db962f3eb1599009570fb116363ac1e63169c00f541a5f21d"}
Nov 03 17:45:06.703890 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:06.703870    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-97p65" event={"ID":"c29d200a-c917-46e5-9cf5-21f4d952e470","Type":"ContainerDied","Data":"a068e219a50a01990dd55ec22bf5723658419d68bca038fbea25f88e779d0e78"}
Nov 03 17:45:06.703934 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:06.703923    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-97p65" event={"ID":"c29d200a-c917-46e5-9cf5-21f4d952e470","Type":"ContainerStarted","Data":"d2a9e78d086aacb77e10555a2cecf7f886eb020720313597be3963fc3700318c"}
Nov 03 17:45:07.708657 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:07.708462    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-97p65_c29d200a-c917-46e5-9cf5-21f4d952e470/ovn-acl-logging/0.log"
Nov 03 17:45:07.709055 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:07.709016    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-97p65" event={"ID":"c29d200a-c917-46e5-9cf5-21f4d952e470","Type":"ContainerStarted","Data":"f0f5459d34b758dfa7ef94e30de7c2eeeba9a550c360efed5727dde008c7dcb6"}
Nov 03 17:45:07.709115 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:07.709098    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-97p65" event={"ID":"c29d200a-c917-46e5-9cf5-21f4d952e470","Type":"ContainerStarted","Data":"4b682f92cd4349b0294ed0cfd69c576db30388f1439eb9e8d6761465eb8d6a60"}
Nov 03 17:45:07.709175 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:07.709155    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-97p65" event={"ID":"c29d200a-c917-46e5-9cf5-21f4d952e470","Type":"ContainerStarted","Data":"7d32ed34059607b57648c02887be1eafafe3a8b1881a6bdc70d0b7679e330b5e"}
Nov 03 17:45:07.711520 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:07.711497    2576 generic.go:356] "Generic (PLEG): container finished" podID="b993b313-db8b-42f5-ad99-2d2ab5bb49e0" containerID="641f560f56ab02e85090e4a58030382c328364cd9caab180caad9fee6767d835" exitCode=0
Nov 03 17:45:07.711649 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:07.711529    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bsgc5" event={"ID":"b993b313-db8b-42f5-ad99-2d2ab5bb49e0","Type":"ContainerDied","Data":"641f560f56ab02e85090e4a58030382c328364cd9caab180caad9fee6767d835"}
Nov 03 17:45:08.556429 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:08.556384    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z264t"
Nov 03 17:45:08.556429 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:08.556419    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xjw2"
Nov 03 17:45:08.556733 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:08.556406    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2t57w"
Nov 03 17:45:08.556733 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:08.556522    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z264t" podUID="9cf65c17-1c79-4cd5-9ef1-dceb467878b0"
Nov 03 17:45:08.556733 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:08.556634    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2t57w" podUID="43c9da28-334d-4bb3-a6bc-e3da2d561633"
Nov 03 17:45:08.556733 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:08.556692    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xjw2" podUID="f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30"
Nov 03 17:45:08.716535 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:08.716505    2576 generic.go:356] "Generic (PLEG): container finished" podID="b993b313-db8b-42f5-ad99-2d2ab5bb49e0" containerID="d2f95a9ec0a2b48e2b53dfa49edb2caff146bbf1357f5793bf45ca6be0372834" exitCode=0
Nov 03 17:45:08.717024 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:08.716551    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bsgc5" event={"ID":"b993b313-db8b-42f5-ad99-2d2ab5bb49e0","Type":"ContainerDied","Data":"d2f95a9ec0a2b48e2b53dfa49edb2caff146bbf1357f5793bf45ca6be0372834"}
Nov 03 17:45:09.724686 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:09.724654    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-97p65_c29d200a-c917-46e5-9cf5-21f4d952e470/ovn-acl-logging/0.log"
Nov 03 17:45:09.725171 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:09.725027    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-97p65" event={"ID":"c29d200a-c917-46e5-9cf5-21f4d952e470","Type":"ContainerStarted","Data":"56363baf4c1eecac23b885def65caf2b5fe7e5edfa4cf4031d397709be5ada14"}
Nov 03 17:45:09.728136 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:09.728115    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bsgc5" event={"ID":"b993b313-db8b-42f5-ad99-2d2ab5bb49e0","Type":"ContainerStarted","Data":"7efdcf830adac2d4d7fc6cded8c9781a3615214b9fa0b8a24b4cf8392fb5e588"}
Nov 03 17:45:09.858360 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:09.858310    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bsgc5" podStartSLOduration=8.579483297 podStartE2EDuration="20.858297098s" totalImagesPullingTime="12.278813801s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:44:49 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:09.857817511 +0000 UTC m=+20.833465252" watchObservedRunningTime="2026-11-03 17:45:09.858297098 +0000 UTC m=+20.833944841"
Nov 03 17:45:10.556298 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:10.556258    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z264t"
Nov 03 17:45:10.556455 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:10.556309    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xjw2"
Nov 03 17:45:10.556455 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:10.556272    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2t57w"
Nov 03 17:45:10.556455 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:10.556394    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z264t" podUID="9cf65c17-1c79-4cd5-9ef1-dceb467878b0"
Nov 03 17:45:10.556580 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:10.556453    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2t57w" podUID="43c9da28-334d-4bb3-a6bc-e3da2d561633"
Nov 03 17:45:10.556580 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:10.556509    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xjw2" podUID="f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30"
Nov 03 17:45:11.735190 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:11.735166    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-97p65_c29d200a-c917-46e5-9cf5-21f4d952e470/ovn-acl-logging/0.log"
Nov 03 17:45:11.735780 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:11.735523    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-97p65" event={"ID":"c29d200a-c917-46e5-9cf5-21f4d952e470","Type":"ContainerStarted","Data":"b61d6227bfff1d37d2b102253c558c7bcae5badfe95fb294eee71cd7a4801ad7"}
Nov 03 17:45:11.735862 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:11.735791    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:45:11.735922 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:11.735906    2576 scope.go:122] "RemoveContainer" containerID="a068e219a50a01990dd55ec22bf5723658419d68bca038fbea25f88e779d0e78"
Nov 03 17:45:11.753384 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:11.753154    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:45:12.556438 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:12.556398    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2t57w"
Nov 03 17:45:12.556438 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:12.556412    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z264t"
Nov 03 17:45:12.556720 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:12.556585    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2t57w" podUID="43c9da28-334d-4bb3-a6bc-e3da2d561633"
Nov 03 17:45:12.556720 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:12.556625    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xjw2"
Nov 03 17:45:12.556797 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:12.556733    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z264t" podUID="9cf65c17-1c79-4cd5-9ef1-dceb467878b0"
Nov 03 17:45:12.556834 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:12.556804    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xjw2" podUID="f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30"
Nov 03 17:45:12.741231 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:12.741200    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-97p65_c29d200a-c917-46e5-9cf5-21f4d952e470/ovn-acl-logging/0.log"
Nov 03 17:45:12.741628 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:12.741528    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-97p65" event={"ID":"c29d200a-c917-46e5-9cf5-21f4d952e470","Type":"ContainerStarted","Data":"6f168334ed973c9f28737a4f00b545eb7c84f42c2e73fc2872af4b656c7760e7"}
Nov 03 17:45:12.741732 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:12.741715    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:45:12.741793 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:12.741783    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:45:12.756958 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:12.756929    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:45:12.924976 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:12.924881    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-97p65" podStartSLOduration=8.52232705 podStartE2EDuration="23.924868496s" totalImagesPullingTime="15.402541446s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:12.924678873 +0000 UTC m=+23.900326617" watchObservedRunningTime="2026-11-03 17:45:12.924868496 +0000 UTC m=+23.900516240"
Nov 03 17:45:13.761313 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:13.761245    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4xjw2"]
Nov 03 17:45:13.761782 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:13.761350    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xjw2"
Nov 03 17:45:13.761782 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:13.761461    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xjw2" podUID="f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30"
Nov 03 17:45:13.764283 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:13.764267    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2t57w"]
Nov 03 17:45:13.764347 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:13.764337    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2t57w"
Nov 03 17:45:13.764419 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:13.764403    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2t57w" podUID="43c9da28-334d-4bb3-a6bc-e3da2d561633"
Nov 03 17:45:13.772126 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:13.772105    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-z264t"]
Nov 03 17:45:13.772243 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:13.772176    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z264t"
Nov 03 17:45:13.772282 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:13.772242    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z264t" podUID="9cf65c17-1c79-4cd5-9ef1-dceb467878b0"
Nov 03 17:45:15.556211 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:15.556121    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z264t"
Nov 03 17:45:15.556211 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:15.556159    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xjw2"
Nov 03 17:45:15.556711 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:15.556121    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2t57w"
Nov 03 17:45:15.556711 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:15.556263    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z264t" podUID="9cf65c17-1c79-4cd5-9ef1-dceb467878b0"
Nov 03 17:45:15.556711 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:15.556365    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2t57w" podUID="43c9da28-334d-4bb3-a6bc-e3da2d561633"
Nov 03 17:45:15.556711 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:15.556430    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xjw2" podUID="f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30"
Nov 03 17:45:15.967659 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:15.967582    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9j547_a831ae97-8f46-4b85-bfdb-59042b7b8b0c/dns-node-resolver/0.log"
Nov 03 17:45:16.765507 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:16.765481    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-x8f2p_79212f04-d10c-43ee-9dd5-41b4fd1917b2/node-ca/0.log"
Nov 03 17:45:17.555759 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:17.555699    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z264t"
Nov 03 17:45:17.555959 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:17.555705    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xjw2"
Nov 03 17:45:17.555959 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:17.555851    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z264t" podUID="9cf65c17-1c79-4cd5-9ef1-dceb467878b0"
Nov 03 17:45:17.555959 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:17.555705    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2t57w"
Nov 03 17:45:17.555959 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:17.555884    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xjw2" podUID="f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30"
Nov 03 17:45:17.555959 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:17.555955    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2t57w" podUID="43c9da28-334d-4bb3-a6bc-e3da2d561633"
Nov 03 17:45:19.557300 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:19.557109    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xjw2"
Nov 03 17:45:19.557773 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:19.557179    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z264t"
Nov 03 17:45:19.557773 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:19.557375    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xjw2" podUID="f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30"
Nov 03 17:45:19.557773 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:19.557206    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2t57w"
Nov 03 17:45:19.557773 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:19.557439    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z264t" podUID="9cf65c17-1c79-4cd5-9ef1-dceb467878b0"
Nov 03 17:45:19.557773 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:19.557514    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2t57w" podUID="43c9da28-334d-4bb3-a6bc-e3da2d561633"
Nov 03 17:45:21.556038 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:21.556009    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2t57w"
Nov 03 17:45:21.556473 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:21.556016    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xjw2"
Nov 03 17:45:21.556473 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:21.556115    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2t57w" podUID="43c9da28-334d-4bb3-a6bc-e3da2d561633"
Nov 03 17:45:21.556473 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:21.556218    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xjw2" podUID="f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30"
Nov 03 17:45:21.556473 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:21.556016    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z264t"
Nov 03 17:45:21.556473 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:21.556313    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z264t" podUID="9cf65c17-1c79-4cd5-9ef1-dceb467878b0"
Nov 03 17:45:22.462835 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:22.462800    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cf65c17-1c79-4cd5-9ef1-dceb467878b0-metrics-certs\") pod \"network-metrics-daemon-z264t\" (UID: \"9cf65c17-1c79-4cd5-9ef1-dceb467878b0\") " pod="openshift-multus/network-metrics-daemon-z264t"
Nov 03 17:45:22.463010 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:22.462842    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30-original-pull-secret\") pod \"global-pull-secret-syncer-4xjw2\" (UID: \"f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30\") " pod="kube-system/global-pull-secret-syncer-4xjw2"
Nov 03 17:45:22.463010 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:22.462929    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 17:45:22.463010 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:22.462936    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:45:22.463010 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:22.462977    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30-original-pull-secret podName:f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:54.462964431 +0000 UTC m=+65.438612165 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30-original-pull-secret") pod "global-pull-secret-syncer-4xjw2" (UID: "f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30") : object "kube-system"/"original-pull-secret" not registered
Nov 03 17:45:22.463010 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:22.462990    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cf65c17-1c79-4cd5-9ef1-dceb467878b0-metrics-certs podName:9cf65c17-1c79-4cd5-9ef1-dceb467878b0 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:54.462983449 +0000 UTC m=+65.438631171 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9cf65c17-1c79-4cd5-9ef1-dceb467878b0-metrics-certs") pod "network-metrics-daemon-z264t" (UID: "9cf65c17-1c79-4cd5-9ef1-dceb467878b0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:45:22.563723 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:22.563690    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v89s6\" (UniqueName: \"kubernetes.io/projected/43c9da28-334d-4bb3-a6bc-e3da2d561633-kube-api-access-v89s6\") pod \"network-check-target-2t57w\" (UID: \"43c9da28-334d-4bb3-a6bc-e3da2d561633\") " pod="openshift-network-diagnostics/network-check-target-2t57w"
Nov 03 17:45:22.564090 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:22.563809    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 17:45:22.564090 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:22.563825    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 17:45:22.564090 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:22.563833    2576 projected.go:196] Error preparing data for projected volume kube-api-access-v89s6 for pod openshift-network-diagnostics/network-check-target-2t57w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:45:22.564090 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:22.563878    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43c9da28-334d-4bb3-a6bc-e3da2d561633-kube-api-access-v89s6 podName:43c9da28-334d-4bb3-a6bc-e3da2d561633 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:54.563865419 +0000 UTC m=+65.539513141 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-v89s6" (UniqueName: "kubernetes.io/projected/43c9da28-334d-4bb3-a6bc-e3da2d561633-kube-api-access-v89s6") pod "network-check-target-2t57w" (UID: "43c9da28-334d-4bb3-a6bc-e3da2d561633") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:45:23.556566 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:23.556529    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z264t"
Nov 03 17:45:23.556566 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:23.556558    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xjw2"
Nov 03 17:45:23.556805 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:23.556655    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2t57w"
Nov 03 17:45:23.556805 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:23.556741    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z264t" podUID="9cf65c17-1c79-4cd5-9ef1-dceb467878b0"
Nov 03 17:45:23.556914 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:23.556886    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xjw2" podUID="f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30"
Nov 03 17:45:23.556997 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:23.556978    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2t57w" podUID="43c9da28-334d-4bb3-a6bc-e3da2d561633"
Nov 03 17:45:25.555876 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:25.555841    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2t57w"
Nov 03 17:45:25.556376 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:25.555885    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xjw2"
Nov 03 17:45:25.556376 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:25.555841    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z264t"
Nov 03 17:45:25.556376 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:25.555958    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2t57w" podUID="43c9da28-334d-4bb3-a6bc-e3da2d561633"
Nov 03 17:45:25.556376 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:25.556073    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xjw2" podUID="f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30"
Nov 03 17:45:25.556376 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:25.556135    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z264t" podUID="9cf65c17-1c79-4cd5-9ef1-dceb467878b0"
Nov 03 17:45:27.556540 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:27.556502    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2t57w"
Nov 03 17:45:27.556540 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:27.556530    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z264t"
Nov 03 17:45:27.556984 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:27.556681    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2t57w" podUID="43c9da28-334d-4bb3-a6bc-e3da2d561633"
Nov 03 17:45:27.556984 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:27.556717    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xjw2"
Nov 03 17:45:27.556984 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:27.556808    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z264t" podUID="9cf65c17-1c79-4cd5-9ef1-dceb467878b0"
Nov 03 17:45:27.556984 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:27.556887    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xjw2" podUID="f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30"
Nov 03 17:45:29.558300 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:29.558263    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z264t"
Nov 03 17:45:29.558996 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:29.558964    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z264t" podUID="9cf65c17-1c79-4cd5-9ef1-dceb467878b0"
Nov 03 17:45:29.559328 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:29.559309    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xjw2"
Nov 03 17:45:29.559726 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:29.559705    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2t57w"
Nov 03 17:45:29.559816 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:29.559726    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xjw2" podUID="f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30"
Nov 03 17:45:29.560121 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:29.560096    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2t57w" podUID="43c9da28-334d-4bb3-a6bc-e3da2d561633"
Nov 03 17:45:31.555495 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:31.555465    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xjw2"
Nov 03 17:45:31.555975 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:31.555479    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z264t"
Nov 03 17:45:31.555975 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:31.555627    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xjw2" podUID="f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30"
Nov 03 17:45:31.555975 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:31.555479    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2t57w"
Nov 03 17:45:31.555975 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:31.555729    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z264t" podUID="9cf65c17-1c79-4cd5-9ef1-dceb467878b0"
Nov 03 17:45:31.555975 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:31.555814    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2t57w" podUID="43c9da28-334d-4bb3-a6bc-e3da2d561633"
Nov 03 17:45:33.555805 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:33.555767    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2t57w"
Nov 03 17:45:33.556367 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:33.555776    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z264t"
Nov 03 17:45:33.556367 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:33.555901    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2t57w" podUID="43c9da28-334d-4bb3-a6bc-e3da2d561633"
Nov 03 17:45:33.556367 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:33.555775    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xjw2"
Nov 03 17:45:33.556367 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:33.555967    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z264t" podUID="9cf65c17-1c79-4cd5-9ef1-dceb467878b0"
Nov 03 17:45:33.556367 ip-10-0-128-204 kubenswrapper[2576]: E1103 17:45:33.556021    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xjw2" podUID="f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30"
Nov 03 17:45:35.239713 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.239684    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-204.us-west-2.compute.internal" event="NodeReady"
Nov 03 17:45:35.240110 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.239801    2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 17:45:35.358625 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.358577    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qmcxl"]
Nov 03 17:45:35.360315 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.360300    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-tfmb2"]
Nov 03 17:45:35.360430 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.360413    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qmcxl"
Nov 03 17:45:35.362017 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.361998    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tfmb2"
Nov 03 17:45:35.402919 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.402849    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tfmb2"]
Nov 03 17:45:35.405709 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.405684    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qmcxl"]
Nov 03 17:45:35.414459 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.414438    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pvbgk\""
Nov 03 17:45:35.414569 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.414517    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 17:45:35.414569 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.414517    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 17:45:35.414569 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.414528    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 17:45:35.442605 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.442574    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pf6z7"]
Nov 03 17:45:35.444218 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.444204    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pf6z7"
Nov 03 17:45:35.451803 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.451780    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9tww\" (UniqueName: \"kubernetes.io/projected/6e0f2b93-0bed-447a-99ee-5b47f3bc09bf-kube-api-access-r9tww\") pod \"ingress-canary-tfmb2\" (UID: \"6e0f2b93-0bed-447a-99ee-5b47f3bc09bf\") " pod="openshift-ingress-canary/ingress-canary-tfmb2"
Nov 03 17:45:35.451923 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.451825    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b8a607f9-6133-4848-bdd6-d48e447cec86-crio-socket\") pod \"insights-runtime-extractor-qmcxl\" (UID: \"b8a607f9-6133-4848-bdd6-d48e447cec86\") " pod="openshift-insights/insights-runtime-extractor-qmcxl"
Nov 03 17:45:35.451923 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.451846    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b8a607f9-6133-4848-bdd6-d48e447cec86-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qmcxl\" (UID: \"b8a607f9-6133-4848-bdd6-d48e447cec86\") " pod="openshift-insights/insights-runtime-extractor-qmcxl"
Nov 03 17:45:35.451923 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.451905    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zn5r\" (UniqueName: \"kubernetes.io/projected/b8a607f9-6133-4848-bdd6-d48e447cec86-kube-api-access-2zn5r\") pod \"insights-runtime-extractor-qmcxl\" (UID: \"b8a607f9-6133-4848-bdd6-d48e447cec86\") " pod="openshift-insights/insights-runtime-extractor-qmcxl"
Nov 03 17:45:35.452073 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.451936    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8a607f9-6133-4848-bdd6-d48e447cec86-data-volume\") pod \"insights-runtime-extractor-qmcxl\" (UID: \"b8a607f9-6133-4848-bdd6-d48e447cec86\") " pod="openshift-insights/insights-runtime-extractor-qmcxl"
Nov 03 17:45:35.452073 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.452014    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b8a607f9-6133-4848-bdd6-d48e447cec86-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qmcxl\" (UID: \"b8a607f9-6133-4848-bdd6-d48e447cec86\") " pod="openshift-insights/insights-runtime-extractor-qmcxl"
Nov 03 17:45:35.452073 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.452033    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e0f2b93-0bed-447a-99ee-5b47f3bc09bf-cert\") pod \"ingress-canary-tfmb2\" (UID: \"6e0f2b93-0bed-447a-99ee-5b47f3bc09bf\") " pod="openshift-ingress-canary/ingress-canary-tfmb2"
Nov 03 17:45:35.504271 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.504236    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pf6z7"]
Nov 03 17:45:35.514880 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.514849    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 17:45:35.515092 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.514943    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 17:45:35.515092 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.514953    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 17:45:35.515092 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.515033    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 17:45:35.515261 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.515244    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 17:45:35.515306 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.515290    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 17:45:35.515347 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.515326    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xfh9g\""
Nov 03 17:45:35.515379 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.515336    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nvxmw\""
Nov 03 17:45:35.553058 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.553019    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b8a607f9-6133-4848-bdd6-d48e447cec86-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qmcxl\" (UID: \"b8a607f9-6133-4848-bdd6-d48e447cec86\") " pod="openshift-insights/insights-runtime-extractor-qmcxl"
Nov 03 17:45:35.553058 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.553060    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zn5r\" (UniqueName: \"kubernetes.io/projected/b8a607f9-6133-4848-bdd6-d48e447cec86-kube-api-access-2zn5r\") pod \"insights-runtime-extractor-qmcxl\" (UID: \"b8a607f9-6133-4848-bdd6-d48e447cec86\") " pod="openshift-insights/insights-runtime-extractor-qmcxl"
Nov 03 17:45:35.553353 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.553091    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e881e23-d118-4245-b855-b1322d133ef1-tmp-dir\") pod \"dns-default-pf6z7\" (UID: \"3e881e23-d118-4245-b855-b1322d133ef1\") " pod="openshift-dns/dns-default-pf6z7"
Nov 03 17:45:35.553353 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.553117    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jsj9\" (UniqueName: \"kubernetes.io/projected/3e881e23-d118-4245-b855-b1322d133ef1-kube-api-access-9jsj9\") pod \"dns-default-pf6z7\" (UID: \"3e881e23-d118-4245-b855-b1322d133ef1\") " pod="openshift-dns/dns-default-pf6z7"
Nov 03 17:45:35.553353 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.553194    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e881e23-d118-4245-b855-b1322d133ef1-metrics-tls\") pod \"dns-default-pf6z7\" (UID: \"3e881e23-d118-4245-b855-b1322d133ef1\") " pod="openshift-dns/dns-default-pf6z7"
Nov 03 17:45:35.553353 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.553225    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8a607f9-6133-4848-bdd6-d48e447cec86-data-volume\") pod \"insights-runtime-extractor-qmcxl\" (UID: \"b8a607f9-6133-4848-bdd6-d48e447cec86\") " pod="openshift-insights/insights-runtime-extractor-qmcxl"
Nov 03 17:45:35.553353 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.553282    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b8a607f9-6133-4848-bdd6-d48e447cec86-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qmcxl\" (UID: \"b8a607f9-6133-4848-bdd6-d48e447cec86\") " pod="openshift-insights/insights-runtime-extractor-qmcxl"
Nov 03 17:45:35.553353 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.553339    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3e881e23-d118-4245-b855-b1322d133ef1-config-volume\") pod \"dns-default-pf6z7\" (UID: \"3e881e23-d118-4245-b855-b1322d133ef1\") " pod="openshift-dns/dns-default-pf6z7"
Nov 03 17:45:35.553560 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.553378    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e0f2b93-0bed-447a-99ee-5b47f3bc09bf-cert\") pod \"ingress-canary-tfmb2\" (UID: \"6e0f2b93-0bed-447a-99ee-5b47f3bc09bf\") " pod="openshift-ingress-canary/ingress-canary-tfmb2"
Nov 03 17:45:35.553560 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.553462    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9tww\" (UniqueName: \"kubernetes.io/projected/6e0f2b93-0bed-447a-99ee-5b47f3bc09bf-kube-api-access-r9tww\") pod \"ingress-canary-tfmb2\" (UID: \"6e0f2b93-0bed-447a-99ee-5b47f3bc09bf\") " pod="openshift-ingress-canary/ingress-canary-tfmb2"
Nov 03 17:45:35.553560 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.553521    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b8a607f9-6133-4848-bdd6-d48e447cec86-crio-socket\") pod \"insights-runtime-extractor-qmcxl\" (UID: \"b8a607f9-6133-4848-bdd6-d48e447cec86\") " pod="openshift-insights/insights-runtime-extractor-qmcxl"
Nov 03 17:45:35.553743 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.553724    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b8a607f9-6133-4848-bdd6-d48e447cec86-crio-socket\") pod \"insights-runtime-extractor-qmcxl\" (UID: \"b8a607f9-6133-4848-bdd6-d48e447cec86\") " pod="openshift-insights/insights-runtime-extractor-qmcxl"
Nov 03 17:45:35.553796 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.553742    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8a607f9-6133-4848-bdd6-d48e447cec86-data-volume\") pod \"insights-runtime-extractor-qmcxl\" (UID: \"b8a607f9-6133-4848-bdd6-d48e447cec86\") " pod="openshift-insights/insights-runtime-extractor-qmcxl"
Nov 03 17:45:35.553970 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.553951    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b8a607f9-6133-4848-bdd6-d48e447cec86-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qmcxl\" (UID: \"b8a607f9-6133-4848-bdd6-d48e447cec86\") " pod="openshift-insights/insights-runtime-extractor-qmcxl"
Nov 03 17:45:35.556089 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.556070    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2t57w"
Nov 03 17:45:35.556174 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.556103    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z264t"
Nov 03 17:45:35.556174 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.556070    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xjw2"
Nov 03 17:45:35.557114 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.557096    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e0f2b93-0bed-447a-99ee-5b47f3bc09bf-cert\") pod \"ingress-canary-tfmb2\" (UID: \"6e0f2b93-0bed-447a-99ee-5b47f3bc09bf\") " pod="openshift-ingress-canary/ingress-canary-tfmb2"
Nov 03 17:45:35.557646 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.557629    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b8a607f9-6133-4848-bdd6-d48e447cec86-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qmcxl\" (UID: \"b8a607f9-6133-4848-bdd6-d48e447cec86\") " pod="openshift-insights/insights-runtime-extractor-qmcxl"
Nov 03 17:45:35.615222 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.615193    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 17:45:35.615380 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.615305    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 17:45:35.615438 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.615380    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f85t9\""
Nov 03 17:45:35.615591 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.615575    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-sq9vz\""
Nov 03 17:45:35.615730 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.615716    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 17:45:35.615797 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.615739    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 17:45:35.615797 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.615763    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9tww\" (UniqueName: \"kubernetes.io/projected/6e0f2b93-0bed-447a-99ee-5b47f3bc09bf-kube-api-access-r9tww\") pod \"ingress-canary-tfmb2\" (UID: \"6e0f2b93-0bed-447a-99ee-5b47f3bc09bf\") " pod="openshift-ingress-canary/ingress-canary-tfmb2"
Nov 03 17:45:35.615797 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.615774    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6lzh9\""
Nov 03 17:45:35.615935 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.615875    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zn5r\" (UniqueName: \"kubernetes.io/projected/b8a607f9-6133-4848-bdd6-d48e447cec86-kube-api-access-2zn5r\") pod \"insights-runtime-extractor-qmcxl\" (UID: \"b8a607f9-6133-4848-bdd6-d48e447cec86\") " pod="openshift-insights/insights-runtime-extractor-qmcxl"
Nov 03 17:45:35.654189 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.654100    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3e881e23-d118-4245-b855-b1322d133ef1-config-volume\") pod \"dns-default-pf6z7\" (UID: \"3e881e23-d118-4245-b855-b1322d133ef1\") " pod="openshift-dns/dns-default-pf6z7"
Nov 03 17:45:35.654189 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.654166    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e881e23-d118-4245-b855-b1322d133ef1-tmp-dir\") pod \"dns-default-pf6z7\" (UID: \"3e881e23-d118-4245-b855-b1322d133ef1\") " pod="openshift-dns/dns-default-pf6z7"
Nov 03 17:45:35.654189 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.654182    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jsj9\" (UniqueName: \"kubernetes.io/projected/3e881e23-d118-4245-b855-b1322d133ef1-kube-api-access-9jsj9\") pod \"dns-default-pf6z7\" (UID: \"3e881e23-d118-4245-b855-b1322d133ef1\") " pod="openshift-dns/dns-default-pf6z7"
Nov 03 17:45:35.654393 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.654198    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e881e23-d118-4245-b855-b1322d133ef1-metrics-tls\") pod \"dns-default-pf6z7\" (UID: \"3e881e23-d118-4245-b855-b1322d133ef1\") " pod="openshift-dns/dns-default-pf6z7"
Nov 03 17:45:35.654585 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.654565    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e881e23-d118-4245-b855-b1322d133ef1-tmp-dir\") pod \"dns-default-pf6z7\" (UID: \"3e881e23-d118-4245-b855-b1322d133ef1\") " pod="openshift-dns/dns-default-pf6z7"
Nov 03 17:45:35.654733 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.654715    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3e881e23-d118-4245-b855-b1322d133ef1-config-volume\") pod \"dns-default-pf6z7\" (UID: \"3e881e23-d118-4245-b855-b1322d133ef1\") " pod="openshift-dns/dns-default-pf6z7"
Nov 03 17:45:35.656405 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.656379    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e881e23-d118-4245-b855-b1322d133ef1-metrics-tls\") pod \"dns-default-pf6z7\" (UID: \"3e881e23-d118-4245-b855-b1322d133ef1\") " pod="openshift-dns/dns-default-pf6z7"
Nov 03 17:45:35.673906 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.673871    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qmcxl"
Nov 03 17:45:35.675819 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.675803    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tfmb2"
Nov 03 17:45:35.717648 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.717599    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jsj9\" (UniqueName: \"kubernetes.io/projected/3e881e23-d118-4245-b855-b1322d133ef1-kube-api-access-9jsj9\") pod \"dns-default-pf6z7\" (UID: \"3e881e23-d118-4245-b855-b1322d133ef1\") " pod="openshift-dns/dns-default-pf6z7"
Nov 03 17:45:35.755748 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:35.755724    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pf6z7"
Nov 03 17:45:36.057454 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:36.057431    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tfmb2"]
Nov 03 17:45:36.060123 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:45:36.060098    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e0f2b93_0bed_447a_99ee_5b47f3bc09bf.slice/crio-7bd829a4c167033d557935b56396132797cbcee937ae350c15682fc836a324b2 WatchSource:0}: Error finding container 7bd829a4c167033d557935b56396132797cbcee937ae350c15682fc836a324b2: Status 404 returned error can't find the container with id 7bd829a4c167033d557935b56396132797cbcee937ae350c15682fc836a324b2
Nov 03 17:45:36.109004 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:36.108979    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qmcxl"]
Nov 03 17:45:36.110329 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:45:36.110307    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8a607f9_6133_4848_bdd6_d48e447cec86.slice/crio-0bb1fb1187d8ea80dd6c6dda38067a5c154fb2c249c4c69851fcc12aab410f85 WatchSource:0}: Error finding container 0bb1fb1187d8ea80dd6c6dda38067a5c154fb2c249c4c69851fcc12aab410f85: Status 404 returned error can't find the container with id 0bb1fb1187d8ea80dd6c6dda38067a5c154fb2c249c4c69851fcc12aab410f85
Nov 03 17:45:36.162501 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:36.162462    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pf6z7"]
Nov 03 17:45:36.164394 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:45:36.164369    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e881e23_d118_4245_b855_b1322d133ef1.slice/crio-139653ececacf7123894d53780a05a97b3a3c808ff74c54de72556fa79a00dfa WatchSource:0}: Error finding container 139653ececacf7123894d53780a05a97b3a3c808ff74c54de72556fa79a00dfa: Status 404 returned error can't find the container with id 139653ececacf7123894d53780a05a97b3a3c808ff74c54de72556fa79a00dfa
Nov 03 17:45:36.794914 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:36.794878    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pf6z7" event={"ID":"3e881e23-d118-4245-b855-b1322d133ef1","Type":"ContainerStarted","Data":"139653ececacf7123894d53780a05a97b3a3c808ff74c54de72556fa79a00dfa"}
Nov 03 17:45:36.796026 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:36.796002    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tfmb2" event={"ID":"6e0f2b93-0bed-447a-99ee-5b47f3bc09bf","Type":"ContainerStarted","Data":"7bd829a4c167033d557935b56396132797cbcee937ae350c15682fc836a324b2"}
Nov 03 17:45:36.797389 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:36.797367    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qmcxl" event={"ID":"b8a607f9-6133-4848-bdd6-d48e447cec86","Type":"ContainerStarted","Data":"079929bf0ce6d9d4c10c3a1c35d43de26b03d4f5ac9654619851f63d980736d6"}
Nov 03 17:45:36.797455 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:36.797410    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qmcxl" event={"ID":"b8a607f9-6133-4848-bdd6-d48e447cec86","Type":"ContainerStarted","Data":"0bb1fb1187d8ea80dd6c6dda38067a5c154fb2c249c4c69851fcc12aab410f85"}
Nov 03 17:45:37.802825 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:37.802786    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qmcxl" event={"ID":"b8a607f9-6133-4848-bdd6-d48e447cec86","Type":"ContainerStarted","Data":"be9d5919f7340e61859b1600bcdcd2f07db2a409a0850cbd82c8db8050ee47d4"}
Nov 03 17:45:37.946035 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:37.945994    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-4wksp"]
Nov 03 17:45:37.948251 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:37.948222    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-4wksp"
Nov 03 17:45:38.059350 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:38.059262    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Nov 03 17:45:38.059502 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:38.059396    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 17:45:38.059572 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:38.059560    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Nov 03 17:45:38.059630 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:38.059604    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 17:45:38.069000 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:38.068967    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b57a39bc-800c-4ea8-a0ef-4a1721786b70-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-4wksp\" (UID: \"b57a39bc-800c-4ea8-a0ef-4a1721786b70\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4wksp"
Nov 03 17:45:38.069166 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:38.069035    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b57a39bc-800c-4ea8-a0ef-4a1721786b70-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-4wksp\" (UID: \"b57a39bc-800c-4ea8-a0ef-4a1721786b70\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4wksp"
Nov 03 17:45:38.069166 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:38.069106    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjnd5\" (UniqueName: \"kubernetes.io/projected/b57a39bc-800c-4ea8-a0ef-4a1721786b70-kube-api-access-fjnd5\") pod \"prometheus-operator-9cd89b74c-4wksp\" (UID: \"b57a39bc-800c-4ea8-a0ef-4a1721786b70\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4wksp"
Nov 03 17:45:38.069282 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:38.069211    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b57a39bc-800c-4ea8-a0ef-4a1721786b70-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-4wksp\" (UID: \"b57a39bc-800c-4ea8-a0ef-4a1721786b70\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4wksp"
Nov 03 17:45:38.110869 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:38.110827    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-5q5nr\""
Nov 03 17:45:38.111046 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:38.110887    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 17:45:38.112083 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:38.112058    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-4wksp"]
Nov 03 17:45:38.170670 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:38.170626    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b57a39bc-800c-4ea8-a0ef-4a1721786b70-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-4wksp\" (UID: \"b57a39bc-800c-4ea8-a0ef-4a1721786b70\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4wksp"
Nov 03 17:45:38.170841 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:38.170711    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b57a39bc-800c-4ea8-a0ef-4a1721786b70-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-4wksp\" (UID: \"b57a39bc-800c-4ea8-a0ef-4a1721786b70\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4wksp"
Nov 03 17:45:38.170841 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:38.170740    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjnd5\" (UniqueName: \"kubernetes.io/projected/b57a39bc-800c-4ea8-a0ef-4a1721786b70-kube-api-access-fjnd5\") pod \"prometheus-operator-9cd89b74c-4wksp\" (UID: \"b57a39bc-800c-4ea8-a0ef-4a1721786b70\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4wksp"
Nov 03 17:45:38.170841 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:38.170791    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b57a39bc-800c-4ea8-a0ef-4a1721786b70-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-4wksp\" (UID: \"b57a39bc-800c-4ea8-a0ef-4a1721786b70\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4wksp"
Nov 03 17:45:38.171414 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:38.171387    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b57a39bc-800c-4ea8-a0ef-4a1721786b70-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-4wksp\" (UID: \"b57a39bc-800c-4ea8-a0ef-4a1721786b70\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4wksp"
Nov 03 17:45:38.173356 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:38.173327    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b57a39bc-800c-4ea8-a0ef-4a1721786b70-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-4wksp\" (UID: \"b57a39bc-800c-4ea8-a0ef-4a1721786b70\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4wksp"
Nov 03 17:45:38.173644 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:38.173627    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b57a39bc-800c-4ea8-a0ef-4a1721786b70-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-4wksp\" (UID: \"b57a39bc-800c-4ea8-a0ef-4a1721786b70\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4wksp"
Nov 03 17:45:38.234697 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:38.234662    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjnd5\" (UniqueName: \"kubernetes.io/projected/b57a39bc-800c-4ea8-a0ef-4a1721786b70-kube-api-access-fjnd5\") pod \"prometheus-operator-9cd89b74c-4wksp\" (UID: \"b57a39bc-800c-4ea8-a0ef-4a1721786b70\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4wksp"
Nov 03 17:45:38.261396 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:38.261360    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-4wksp"
Nov 03 17:45:38.637489 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:38.637399    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-4wksp"]
Nov 03 17:45:38.675174 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:45:38.675128    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb57a39bc_800c_4ea8_a0ef_4a1721786b70.slice/crio-20eb0bd6f5eabf540587d2a2680526d0bf218614cb1ae6e6466f48b069f0dfad WatchSource:0}: Error finding container 20eb0bd6f5eabf540587d2a2680526d0bf218614cb1ae6e6466f48b069f0dfad: Status 404 returned error can't find the container with id 20eb0bd6f5eabf540587d2a2680526d0bf218614cb1ae6e6466f48b069f0dfad
Nov 03 17:45:38.807408 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:38.807366    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-4wksp" event={"ID":"b57a39bc-800c-4ea8-a0ef-4a1721786b70","Type":"ContainerStarted","Data":"20eb0bd6f5eabf540587d2a2680526d0bf218614cb1ae6e6466f48b069f0dfad"}
Nov 03 17:45:39.813750 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:39.813710    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qmcxl" event={"ID":"b8a607f9-6133-4848-bdd6-d48e447cec86","Type":"ContainerStarted","Data":"b652cc047dcf9cede563e594d096b9e340f718417a8f56367b7358bec35f70bd"}
Nov 03 17:45:39.815875 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:39.815843    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tfmb2" event={"ID":"6e0f2b93-0bed-447a-99ee-5b47f3bc09bf","Type":"ContainerStarted","Data":"74c83ec0ed8914c7c32b1d64dfed697bf71294d5495abcae81a828911871bd0d"}
Nov 03 17:45:40.821525 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:40.821488    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-4wksp" event={"ID":"b57a39bc-800c-4ea8-a0ef-4a1721786b70","Type":"ContainerStarted","Data":"e28fa46955c984ca9f4d19cf7dcadd23acae11991e80198c3878f51dc1e79c5f"}
Nov 03 17:45:40.821995 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:40.821546    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-4wksp" event={"ID":"b57a39bc-800c-4ea8-a0ef-4a1721786b70","Type":"ContainerStarted","Data":"afb79d92b48fd29a91591547e958ed1ec75e5a71f7731eb50d39553ac315fc0d"}
Nov 03 17:45:40.823397 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:40.823366    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pf6z7" event={"ID":"3e881e23-d118-4245-b855-b1322d133ef1","Type":"ContainerStarted","Data":"cacf248dda0700ff74c8afff68447d36a30d6d5b83f601a70805d753a3052bdd"}
Nov 03 17:45:40.823518 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:40.823423    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pf6z7" event={"ID":"3e881e23-d118-4245-b855-b1322d133ef1","Type":"ContainerStarted","Data":"35ccd3a018143b832fe07d3c62c27bb30dc0997561309ad16feb86afb306fb2e"}
Nov 03 17:45:40.823597 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:40.823579    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pf6z7"
Nov 03 17:45:40.890708 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:40.888643    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-qmcxl" podStartSLOduration=2.706938428 podStartE2EDuration="5.888623985s" totalImagesPullingTime="3.181685557s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:45:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:40.887024239 +0000 UTC m=+51.862671975" watchObservedRunningTime="2026-11-03 17:45:40.888623985 +0000 UTC m=+51.864271723"
Nov 03 17:45:41.006213 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:41.006152    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-tfmb2" podStartSLOduration=2.651288917 podStartE2EDuration="6.006135217s" totalImagesPullingTime="3.3548463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:45:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:41.005700181 +0000 UTC m=+51.981347927" watchObservedRunningTime="2026-11-03 17:45:41.006135217 +0000 UTC m=+51.981782962"
Nov 03 17:45:41.128809 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:41.128709    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-pf6z7" podStartSLOduration=2.367125615 podStartE2EDuration="6.128694065s" totalImagesPullingTime="3.76156845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:45:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:41.127753362 +0000 UTC m=+52.103401106" watchObservedRunningTime="2026-11-03 17:45:41.128694065 +0000 UTC m=+52.104341847"
Nov 03 17:45:41.248232 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:41.248179    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-4wksp" podStartSLOduration=2.486356096 podStartE2EDuration="4.248164277s" totalImagesPullingTime="1.761808181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:45:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:41.247929254 +0000 UTC m=+52.223576998" watchObservedRunningTime="2026-11-03 17:45:41.248164277 +0000 UTC m=+52.223812021"
Nov 03 17:45:44.522695 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:44.522664    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7w9wb"]
Nov 03 17:45:44.524566 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:44.524550    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7w9wb"
Nov 03 17:45:44.619305 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:44.619266    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/55b54081-644c-464e-91e1-15c2d8f5be59-node-exporter-textfile\") pod \"node-exporter-7w9wb\" (UID: \"55b54081-644c-464e-91e1-15c2d8f5be59\") " pod="openshift-monitoring/node-exporter-7w9wb"
Nov 03 17:45:44.619583 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:44.619325    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jstr\" (UniqueName: \"kubernetes.io/projected/55b54081-644c-464e-91e1-15c2d8f5be59-kube-api-access-2jstr\") pod \"node-exporter-7w9wb\" (UID: \"55b54081-644c-464e-91e1-15c2d8f5be59\") " pod="openshift-monitoring/node-exporter-7w9wb"
Nov 03 17:45:44.619583 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:44.619401    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/55b54081-644c-464e-91e1-15c2d8f5be59-metrics-client-ca\") pod \"node-exporter-7w9wb\" (UID: \"55b54081-644c-464e-91e1-15c2d8f5be59\") " pod="openshift-monitoring/node-exporter-7w9wb"
Nov 03 17:45:44.619583 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:44.619465    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/55b54081-644c-464e-91e1-15c2d8f5be59-sys\") pod \"node-exporter-7w9wb\" (UID: \"55b54081-644c-464e-91e1-15c2d8f5be59\") " pod="openshift-monitoring/node-exporter-7w9wb"
Nov 03 17:45:44.619583 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:44.619493    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/55b54081-644c-464e-91e1-15c2d8f5be59-node-exporter-tls\") pod \"node-exporter-7w9wb\" (UID: \"55b54081-644c-464e-91e1-15c2d8f5be59\") " pod="openshift-monitoring/node-exporter-7w9wb"
Nov 03 17:45:44.619583 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:44.619569    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/55b54081-644c-464e-91e1-15c2d8f5be59-node-exporter-accelerators-collector-config\") pod \"node-exporter-7w9wb\" (UID: \"55b54081-644c-464e-91e1-15c2d8f5be59\") " pod="openshift-monitoring/node-exporter-7w9wb"
Nov 03 17:45:44.619782 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:44.619595    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/55b54081-644c-464e-91e1-15c2d8f5be59-node-exporter-wtmp\") pod \"node-exporter-7w9wb\" (UID: \"55b54081-644c-464e-91e1-15c2d8f5be59\") " pod="openshift-monitoring/node-exporter-7w9wb"
Nov 03 17:45:44.619782 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:44.619626    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/55b54081-644c-464e-91e1-15c2d8f5be59-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7w9wb\" (UID: \"55b54081-644c-464e-91e1-15c2d8f5be59\") " pod="openshift-monitoring/node-exporter-7w9wb"
Nov 03 17:45:44.619782 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:44.619667    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/55b54081-644c-464e-91e1-15c2d8f5be59-root\") pod \"node-exporter-7w9wb\" (UID: \"55b54081-644c-464e-91e1-15c2d8f5be59\") " pod="openshift-monitoring/node-exporter-7w9wb"
Nov 03 17:45:44.653346 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:44.653318    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 17:45:44.695888 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:44.695862    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-556wq\""
Nov 03 17:45:44.695888 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:44.695859    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 17:45:44.696049 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:44.695946    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 17:45:44.720175 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:44.720150    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/55b54081-644c-464e-91e1-15c2d8f5be59-sys\") pod \"node-exporter-7w9wb\" (UID: \"55b54081-644c-464e-91e1-15c2d8f5be59\") " pod="openshift-monitoring/node-exporter-7w9wb"
Nov 03 17:45:44.720257 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:44.720179    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/55b54081-644c-464e-91e1-15c2d8f5be59-node-exporter-tls\") pod \"node-exporter-7w9wb\" (UID: \"55b54081-644c-464e-91e1-15c2d8f5be59\") " pod="openshift-monitoring/node-exporter-7w9wb"
Nov 03 17:45:44.720257 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:44.720209    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/55b54081-644c-464e-91e1-15c2d8f5be59-node-exporter-accelerators-collector-config\") pod \"node-exporter-7w9wb\" (UID: \"55b54081-644c-464e-91e1-15c2d8f5be59\") " pod="openshift-monitoring/node-exporter-7w9wb"
Nov 03 17:45:44.720257 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:44.720233    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/55b54081-644c-464e-91e1-15c2d8f5be59-node-exporter-wtmp\") pod \"node-exporter-7w9wb\" (UID: \"55b54081-644c-464e-91e1-15c2d8f5be59\") " pod="openshift-monitoring/node-exporter-7w9wb"
Nov 03 17:45:44.720257 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:44.720250    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/55b54081-644c-464e-91e1-15c2d8f5be59-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7w9wb\" (UID: \"55b54081-644c-464e-91e1-15c2d8f5be59\") " pod="openshift-monitoring/node-exporter-7w9wb"
Nov 03 17:45:44.720441 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:44.720267    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/55b54081-644c-464e-91e1-15c2d8f5be59-sys\") pod \"node-exporter-7w9wb\" (UID: \"55b54081-644c-464e-91e1-15c2d8f5be59\") " pod="openshift-monitoring/node-exporter-7w9wb"
Nov 03 17:45:44.720441 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:44.720271    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/55b54081-644c-464e-91e1-15c2d8f5be59-root\") pod \"node-exporter-7w9wb\" (UID: \"55b54081-644c-464e-91e1-15c2d8f5be59\") " pod="openshift-monitoring/node-exporter-7w9wb"
Nov 03 17:45:44.720441 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:44.720317    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/55b54081-644c-464e-91e1-15c2d8f5be59-root\") pod \"node-exporter-7w9wb\" (UID: \"55b54081-644c-464e-91e1-15c2d8f5be59\") " pod="openshift-monitoring/node-exporter-7w9wb"
Nov 03 17:45:44.720441 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:44.720318    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/55b54081-644c-464e-91e1-15c2d8f5be59-node-exporter-textfile\") pod \"node-exporter-7w9wb\" (UID: \"55b54081-644c-464e-91e1-15c2d8f5be59\") " pod="openshift-monitoring/node-exporter-7w9wb"
Nov 03 17:45:44.720441 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:44.720392    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/55b54081-644c-464e-91e1-15c2d8f5be59-node-exporter-wtmp\") pod \"node-exporter-7w9wb\" (UID: \"55b54081-644c-464e-91e1-15c2d8f5be59\") " pod="openshift-monitoring/node-exporter-7w9wb"
Nov 03 17:45:44.720441 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:44.720400    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jstr\" (UniqueName: \"kubernetes.io/projected/55b54081-644c-464e-91e1-15c2d8f5be59-kube-api-access-2jstr\") pod \"node-exporter-7w9wb\" (UID: \"55b54081-644c-464e-91e1-15c2d8f5be59\") " pod="openshift-monitoring/node-exporter-7w9wb"
Nov 03 17:45:44.720742 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:44.720457    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/55b54081-644c-464e-91e1-15c2d8f5be59-metrics-client-ca\") pod \"node-exporter-7w9wb\" (UID: \"55b54081-644c-464e-91e1-15c2d8f5be59\") " pod="openshift-monitoring/node-exporter-7w9wb"
Nov 03 17:45:44.720742 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:44.720718    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/55b54081-644c-464e-91e1-15c2d8f5be59-node-exporter-textfile\") pod \"node-exporter-7w9wb\" (UID: \"55b54081-644c-464e-91e1-15c2d8f5be59\") " pod="openshift-monitoring/node-exporter-7w9wb"
Nov 03 17:45:44.720880 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:44.720857    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/55b54081-644c-464e-91e1-15c2d8f5be59-node-exporter-accelerators-collector-config\") pod \"node-exporter-7w9wb\" (UID: \"55b54081-644c-464e-91e1-15c2d8f5be59\") " pod="openshift-monitoring/node-exporter-7w9wb"
Nov 03 17:45:44.720962 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:44.720945    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/55b54081-644c-464e-91e1-15c2d8f5be59-metrics-client-ca\") pod \"node-exporter-7w9wb\" (UID: \"55b54081-644c-464e-91e1-15c2d8f5be59\") " pod="openshift-monitoring/node-exporter-7w9wb"
Nov 03 17:45:44.722501 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:44.722482    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/55b54081-644c-464e-91e1-15c2d8f5be59-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7w9wb\" (UID: \"55b54081-644c-464e-91e1-15c2d8f5be59\") " pod="openshift-monitoring/node-exporter-7w9wb"
Nov 03 17:45:44.722584 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:44.722569    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/55b54081-644c-464e-91e1-15c2d8f5be59-node-exporter-tls\") pod \"node-exporter-7w9wb\" (UID: \"55b54081-644c-464e-91e1-15c2d8f5be59\") " pod="openshift-monitoring/node-exporter-7w9wb"
Nov 03 17:45:44.755636 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:44.755597    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:45:44.783928 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:44.783910    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jstr\" (UniqueName: \"kubernetes.io/projected/55b54081-644c-464e-91e1-15c2d8f5be59-kube-api-access-2jstr\") pod \"node-exporter-7w9wb\" (UID: \"55b54081-644c-464e-91e1-15c2d8f5be59\") " pod="openshift-monitoring/node-exporter-7w9wb"
Nov 03 17:45:44.835990 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:44.835963    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7w9wb"
Nov 03 17:45:44.847427 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:45:44.847388    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55b54081_644c_464e_91e1_15c2d8f5be59.slice/crio-919fb8112573214d30fe24a1e5472e0723bcc1c57c5a2f3b98c69c3de9573523 WatchSource:0}: Error finding container 919fb8112573214d30fe24a1e5472e0723bcc1c57c5a2f3b98c69c3de9573523: Status 404 returned error can't find the container with id 919fb8112573214d30fe24a1e5472e0723bcc1c57c5a2f3b98c69c3de9573523
Nov 03 17:45:45.842352 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:45.842317    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7w9wb" event={"ID":"55b54081-644c-464e-91e1-15c2d8f5be59","Type":"ContainerStarted","Data":"919fb8112573214d30fe24a1e5472e0723bcc1c57c5a2f3b98c69c3de9573523"}
Nov 03 17:45:46.846928 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:46.846896    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7w9wb" event={"ID":"55b54081-644c-464e-91e1-15c2d8f5be59","Type":"ContainerStarted","Data":"46b0f51171181de71c40b397b90fde72284af2e29fb6d4d4729af48dca5be168"}
Nov 03 17:45:47.280117 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:47.280082    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-74479fcb5-w59vr"]
Nov 03 17:45:47.282348 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:47.282331    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr"
Nov 03 17:45:47.336847 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:47.336815    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 03 17:45:47.337008 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:47.336988    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-xfqrm\""
Nov 03 17:45:47.337078 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:47.337044    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 03 17:45:47.337078 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:47.337050    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 03 17:45:47.337078 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:47.337075    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-9v3v2ob5pcs9m\""
Nov 03 17:45:47.337195 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:47.337092    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 03 17:45:47.339849 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:47.339834    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 03 17:45:47.346021 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:47.345999    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-74479fcb5-w59vr"]
Nov 03 17:45:47.439938 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:47.439905    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d505ffde-320b-41c5-a02e-374ee52e6a1a-secret-thanos-querier-tls\") pod \"thanos-querier-74479fcb5-w59vr\" (UID: \"d505ffde-320b-41c5-a02e-374ee52e6a1a\") " pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr"
Nov 03 17:45:47.439938 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:47.439946    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d505ffde-320b-41c5-a02e-374ee52e6a1a-secret-grpc-tls\") pod \"thanos-querier-74479fcb5-w59vr\" (UID: \"d505ffde-320b-41c5-a02e-374ee52e6a1a\") " pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr"
Nov 03 17:45:47.440156 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:47.439996    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d505ffde-320b-41c5-a02e-374ee52e6a1a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-74479fcb5-w59vr\" (UID: \"d505ffde-320b-41c5-a02e-374ee52e6a1a\") " pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr"
Nov 03 17:45:47.440156 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:47.440076    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d505ffde-320b-41c5-a02e-374ee52e6a1a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-74479fcb5-w59vr\" (UID: \"d505ffde-320b-41c5-a02e-374ee52e6a1a\") " pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr"
Nov 03 17:45:47.440156 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:47.440129    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d505ffde-320b-41c5-a02e-374ee52e6a1a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-74479fcb5-w59vr\" (UID: \"d505ffde-320b-41c5-a02e-374ee52e6a1a\") " pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr"
Nov 03 17:45:47.440246 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:47.440206    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d505ffde-320b-41c5-a02e-374ee52e6a1a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-74479fcb5-w59vr\" (UID: \"d505ffde-320b-41c5-a02e-374ee52e6a1a\") " pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr"
Nov 03 17:45:47.440246 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:47.440236    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njbgq\" (UniqueName: \"kubernetes.io/projected/d505ffde-320b-41c5-a02e-374ee52e6a1a-kube-api-access-njbgq\") pod \"thanos-querier-74479fcb5-w59vr\" (UID: \"d505ffde-320b-41c5-a02e-374ee52e6a1a\") " pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr"
Nov 03 17:45:47.440308 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:47.440297    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d505ffde-320b-41c5-a02e-374ee52e6a1a-tmp\") pod \"thanos-querier-74479fcb5-w59vr\" (UID: \"d505ffde-320b-41c5-a02e-374ee52e6a1a\") " pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr"
Nov 03 17:45:47.440352 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:47.440339    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d505ffde-320b-41c5-a02e-374ee52e6a1a-metrics-client-ca\") pod \"thanos-querier-74479fcb5-w59vr\" (UID: \"d505ffde-320b-41c5-a02e-374ee52e6a1a\") " pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr"
Nov 03 17:45:47.541087 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:47.541006    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d505ffde-320b-41c5-a02e-374ee52e6a1a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-74479fcb5-w59vr\" (UID: \"d505ffde-320b-41c5-a02e-374ee52e6a1a\") " pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr"
Nov 03 17:45:47.541087 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:47.541039    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njbgq\" (UniqueName: \"kubernetes.io/projected/d505ffde-320b-41c5-a02e-374ee52e6a1a-kube-api-access-njbgq\") pod \"thanos-querier-74479fcb5-w59vr\" (UID: \"d505ffde-320b-41c5-a02e-374ee52e6a1a\") " pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr"
Nov 03 17:45:47.541087 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:47.541060    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d505ffde-320b-41c5-a02e-374ee52e6a1a-tmp\") pod \"thanos-querier-74479fcb5-w59vr\" (UID: \"d505ffde-320b-41c5-a02e-374ee52e6a1a\") " pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr"
Nov 03 17:45:47.541087 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:47.541083    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d505ffde-320b-41c5-a02e-374ee52e6a1a-metrics-client-ca\") pod \"thanos-querier-74479fcb5-w59vr\" (UID: \"d505ffde-320b-41c5-a02e-374ee52e6a1a\") " pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr"
Nov 03 17:45:47.541381 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:47.541111    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d505ffde-320b-41c5-a02e-374ee52e6a1a-secret-thanos-querier-tls\") pod \"thanos-querier-74479fcb5-w59vr\" (UID: \"d505ffde-320b-41c5-a02e-374ee52e6a1a\") " pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr"
Nov 03 17:45:47.541381 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:47.541130    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d505ffde-320b-41c5-a02e-374ee52e6a1a-secret-grpc-tls\") pod \"thanos-querier-74479fcb5-w59vr\" (UID: \"d505ffde-320b-41c5-a02e-374ee52e6a1a\") " pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr"
Nov 03 17:45:47.541381 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:47.541172    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d505ffde-320b-41c5-a02e-374ee52e6a1a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-74479fcb5-w59vr\" (UID: \"d505ffde-320b-41c5-a02e-374ee52e6a1a\") " pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr"
Nov 03 17:45:47.541381 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:47.541202    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d505ffde-320b-41c5-a02e-374ee52e6a1a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-74479fcb5-w59vr\" (UID: \"d505ffde-320b-41c5-a02e-374ee52e6a1a\") " pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr"
Nov 03 17:45:47.541381 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:47.541249    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d505ffde-320b-41c5-a02e-374ee52e6a1a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-74479fcb5-w59vr\" (UID: \"d505ffde-320b-41c5-a02e-374ee52e6a1a\") " pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr"
Nov 03 17:45:47.541649 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:47.541491    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d505ffde-320b-41c5-a02e-374ee52e6a1a-tmp\") pod \"thanos-querier-74479fcb5-w59vr\" (UID: \"d505ffde-320b-41c5-a02e-374ee52e6a1a\") " pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr"
Nov 03 17:45:47.542003 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:47.541971    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d505ffde-320b-41c5-a02e-374ee52e6a1a-metrics-client-ca\") pod \"thanos-querier-74479fcb5-w59vr\" (UID: \"d505ffde-320b-41c5-a02e-374ee52e6a1a\") " pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr"
Nov 03 17:45:47.546191 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:47.544282    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d505ffde-320b-41c5-a02e-374ee52e6a1a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-74479fcb5-w59vr\" (UID: \"d505ffde-320b-41c5-a02e-374ee52e6a1a\") " pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr"
Nov 03 17:45:47.546191 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:47.544326    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d505ffde-320b-41c5-a02e-374ee52e6a1a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-74479fcb5-w59vr\" (UID: \"d505ffde-320b-41c5-a02e-374ee52e6a1a\") " pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr"
Nov 03 17:45:47.546191 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:47.544438    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d505ffde-320b-41c5-a02e-374ee52e6a1a-secret-grpc-tls\") pod \"thanos-querier-74479fcb5-w59vr\" (UID: \"d505ffde-320b-41c5-a02e-374ee52e6a1a\") " pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr"
Nov 03 17:45:47.546191 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:47.544587    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d505ffde-320b-41c5-a02e-374ee52e6a1a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-74479fcb5-w59vr\" (UID: \"d505ffde-320b-41c5-a02e-374ee52e6a1a\") " pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr"
Nov 03 17:45:47.546191 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:47.544652    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d505ffde-320b-41c5-a02e-374ee52e6a1a-secret-thanos-querier-tls\") pod \"thanos-querier-74479fcb5-w59vr\" (UID: \"d505ffde-320b-41c5-a02e-374ee52e6a1a\") " pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr"
Nov 03 17:45:47.546191 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:47.544971    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d505ffde-320b-41c5-a02e-374ee52e6a1a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-74479fcb5-w59vr\" (UID: \"d505ffde-320b-41c5-a02e-374ee52e6a1a\") " pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr"
Nov 03 17:45:47.603785 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:47.603755    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njbgq\" (UniqueName: \"kubernetes.io/projected/d505ffde-320b-41c5-a02e-374ee52e6a1a-kube-api-access-njbgq\") pod \"thanos-querier-74479fcb5-w59vr\" (UID: \"d505ffde-320b-41c5-a02e-374ee52e6a1a\") " pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr"
Nov 03 17:45:47.852361 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:47.852258    2576 generic.go:356] "Generic (PLEG): container finished" podID="55b54081-644c-464e-91e1-15c2d8f5be59" containerID="46b0f51171181de71c40b397b90fde72284af2e29fb6d4d4729af48dca5be168" exitCode=0
Nov 03 17:45:47.852731 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:47.852348    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7w9wb" event={"ID":"55b54081-644c-464e-91e1-15c2d8f5be59","Type":"ContainerDied","Data":"46b0f51171181de71c40b397b90fde72284af2e29fb6d4d4729af48dca5be168"}
Nov 03 17:45:47.852731 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:47.852412    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7w9wb" event={"ID":"55b54081-644c-464e-91e1-15c2d8f5be59","Type":"ContainerStarted","Data":"29f60c2a190176ae74b7da918265b2767e27865dcf08cb7f723d589de169902e"}
Nov 03 17:45:47.852731 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:47.852448    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7w9wb" event={"ID":"55b54081-644c-464e-91e1-15c2d8f5be59","Type":"ContainerStarted","Data":"87fedc5dc90a84a2b933ea2a0afc51a2a4c87c688637e0c390bdcd06e71e39b3"}
Nov 03 17:45:47.896536 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:47.896504    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr"
Nov 03 17:45:47.976415 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:47.976368    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7w9wb" podStartSLOduration=2.054200061 podStartE2EDuration="3.976353051s" totalImagesPullingTime="1.85182941s" totalInitContainerRuntime="70.32358ms" isStatefulPod=true podCreationTimestamp="2026-11-03 17:45:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:47.975782862 +0000 UTC m=+58.951430606" watchObservedRunningTime="2026-11-03 17:45:47.976353051 +0000 UTC m=+58.952000795"
Nov 03 17:45:48.324378 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:48.323243    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-74479fcb5-w59vr"]
Nov 03 17:45:48.327672 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:45:48.327644    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd505ffde_320b_41c5_a02e_374ee52e6a1a.slice/crio-a493b902a49a44f6003be191f76a551ca141c08c0d1ece763fbf1d170f15e03b WatchSource:0}: Error finding container a493b902a49a44f6003be191f76a551ca141c08c0d1ece763fbf1d170f15e03b: Status 404 returned error can't find the container with id a493b902a49a44f6003be191f76a551ca141c08c0d1ece763fbf1d170f15e03b
Nov 03 17:45:48.856797 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:48.856755    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr" event={"ID":"d505ffde-320b-41c5-a02e-374ee52e6a1a","Type":"ContainerStarted","Data":"a493b902a49a44f6003be191f76a551ca141c08c0d1ece763fbf1d170f15e03b"}
Nov 03 17:45:49.249944 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:49.249861    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-dwxs2"]
Nov 03 17:45:49.252437 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:49.252421    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-dwxs2"
Nov 03 17:45:49.309304 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:49.309271    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-dwxs2"]
Nov 03 17:45:49.357967 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:49.357925    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cztkv\" (UniqueName: \"kubernetes.io/projected/e106ef93-623a-4cb4-9848-31087f1a7711-kube-api-access-cztkv\") pod \"monitoring-plugin-77f4f76594-dwxs2\" (UID: \"e106ef93-623a-4cb4-9848-31087f1a7711\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-dwxs2"
Nov 03 17:45:49.358149 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:49.358049    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e106ef93-623a-4cb4-9848-31087f1a7711-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-dwxs2\" (UID: \"e106ef93-623a-4cb4-9848-31087f1a7711\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-dwxs2"
Nov 03 17:45:49.359509 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:49.359488    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-mtvgp\""
Nov 03 17:45:49.359631 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:49.359544    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 03 17:45:49.458953 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:49.458917    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cztkv\" (UniqueName: \"kubernetes.io/projected/e106ef93-623a-4cb4-9848-31087f1a7711-kube-api-access-cztkv\") pod \"monitoring-plugin-77f4f76594-dwxs2\" (UID: \"e106ef93-623a-4cb4-9848-31087f1a7711\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-dwxs2"
Nov 03 17:45:49.459123 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:49.458997    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e106ef93-623a-4cb4-9848-31087f1a7711-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-dwxs2\" (UID: \"e106ef93-623a-4cb4-9848-31087f1a7711\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-dwxs2"
Nov 03 17:45:49.512351 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:49.512272    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 03 17:45:49.521929 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:49.521893    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e106ef93-623a-4cb4-9848-31087f1a7711-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-dwxs2\" (UID: \"e106ef93-623a-4cb4-9848-31087f1a7711\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-dwxs2"
Nov 03 17:45:49.522076 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:49.521971    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cztkv\" (UniqueName: \"kubernetes.io/projected/e106ef93-623a-4cb4-9848-31087f1a7711-kube-api-access-cztkv\") pod \"monitoring-plugin-77f4f76594-dwxs2\" (UID: \"e106ef93-623a-4cb4-9848-31087f1a7711\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-dwxs2"
Nov 03 17:45:49.615059 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:49.615022    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-mtvgp\""
Nov 03 17:45:49.623153 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:49.623126    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-dwxs2"
Nov 03 17:45:49.998121 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:49.998089    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-dwxs2"]
Nov 03 17:45:50.001030 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:45:50.000980    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode106ef93_623a_4cb4_9848_31087f1a7711.slice/crio-68856dcdd260964150b8ee215612e33d4f82033dbfa90a19ac4dbea63d6c3330 WatchSource:0}: Error finding container 68856dcdd260964150b8ee215612e33d4f82033dbfa90a19ac4dbea63d6c3330: Status 404 returned error can't find the container with id 68856dcdd260964150b8ee215612e33d4f82033dbfa90a19ac4dbea63d6c3330
Nov 03 17:45:50.829287 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:50.829256    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pf6z7"
Nov 03 17:45:50.864379 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:50.864340    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-dwxs2" event={"ID":"e106ef93-623a-4cb4-9848-31087f1a7711","Type":"ContainerStarted","Data":"68856dcdd260964150b8ee215612e33d4f82033dbfa90a19ac4dbea63d6c3330"}
Nov 03 17:45:51.872474 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:51.872438    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr" event={"ID":"d505ffde-320b-41c5-a02e-374ee52e6a1a","Type":"ContainerStarted","Data":"52ddf7c2280d5908e4520194350b01fbc96f62c20bb159db1fdc178a2013557f"}
Nov 03 17:45:51.872890 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:51.872506    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr" event={"ID":"d505ffde-320b-41c5-a02e-374ee52e6a1a","Type":"ContainerStarted","Data":"e7ca1727bdd683958c0ba278b57bbf1075949051d7b8b6329f29de4ede5c355e"}
Nov 03 17:45:51.872890 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:51.872533    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr" event={"ID":"d505ffde-320b-41c5-a02e-374ee52e6a1a","Type":"ContainerStarted","Data":"ddd235836a270804d7cf63fe75eb99c159a615e241eb5d28a05247ee915ca1f2"}
Nov 03 17:45:53.886547 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:53.885767    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr" event={"ID":"d505ffde-320b-41c5-a02e-374ee52e6a1a","Type":"ContainerStarted","Data":"22da0eff2a5f3040cb3f046539a5c040e9ef458a2088acce08c407d185b50c77"}
Nov 03 17:45:53.886547 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:53.885856    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr" event={"ID":"d505ffde-320b-41c5-a02e-374ee52e6a1a","Type":"ContainerStarted","Data":"1d545132055d4252b2fa27dae93deed82cc05cdd39157dd2ba6aa00a80328430"}
Nov 03 17:45:53.886547 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:53.885898    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr" event={"ID":"d505ffde-320b-41c5-a02e-374ee52e6a1a","Type":"ContainerStarted","Data":"9a2dff1c80a9df710fd07bb4a6df1783ec3d7ec58241712fa8b0f0a939cf6653"}
Nov 03 17:45:53.886547 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:53.885977    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr"
Nov 03 17:45:54.116502 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:54.116443    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr" podStartSLOduration=2.690254728 podStartE2EDuration="7.116424446s" totalImagesPullingTime="4.426169718s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:45:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:54.064393771 +0000 UTC m=+65.040041525" watchObservedRunningTime="2026-11-03 17:45:54.116424446 +0000 UTC m=+65.092072191"
Nov 03 17:45:54.500919 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:54.500877    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cf65c17-1c79-4cd5-9ef1-dceb467878b0-metrics-certs\") pod \"network-metrics-daemon-z264t\" (UID: \"9cf65c17-1c79-4cd5-9ef1-dceb467878b0\") " pod="openshift-multus/network-metrics-daemon-z264t"
Nov 03 17:45:54.501098 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:54.501048    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30-original-pull-secret\") pod \"global-pull-secret-syncer-4xjw2\" (UID: \"f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30\") " pod="kube-system/global-pull-secret-syncer-4xjw2"
Nov 03 17:45:54.554227 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:54.554195    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 17:45:54.554227 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:54.554195    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 17:45:54.563670 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:54.563644    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30-original-pull-secret\") pod \"global-pull-secret-syncer-4xjw2\" (UID: \"f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30\") " pod="kube-system/global-pull-secret-syncer-4xjw2"
Nov 03 17:45:54.563670 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:54.563654    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cf65c17-1c79-4cd5-9ef1-dceb467878b0-metrics-certs\") pod \"network-metrics-daemon-z264t\" (UID: \"9cf65c17-1c79-4cd5-9ef1-dceb467878b0\") " pod="openshift-multus/network-metrics-daemon-z264t"
Nov 03 17:45:54.602055 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:54.601974    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v89s6\" (UniqueName: \"kubernetes.io/projected/43c9da28-334d-4bb3-a6bc-e3da2d561633-kube-api-access-v89s6\") pod \"network-check-target-2t57w\" (UID: \"43c9da28-334d-4bb3-a6bc-e3da2d561633\") " pod="openshift-network-diagnostics/network-check-target-2t57w"
Nov 03 17:45:54.655067 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:54.655030    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 17:45:54.715990 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:54.715955    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 17:45:54.726724 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:54.726690    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v89s6\" (UniqueName: \"kubernetes.io/projected/43c9da28-334d-4bb3-a6bc-e3da2d561633-kube-api-access-v89s6\") pod \"network-check-target-2t57w\" (UID: \"43c9da28-334d-4bb3-a6bc-e3da2d561633\") " pod="openshift-network-diagnostics/network-check-target-2t57w"
Nov 03 17:45:54.828796 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:54.828756    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-sq9vz\""
Nov 03 17:45:54.834325 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:54.834298    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6lzh9\""
Nov 03 17:45:54.835344 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:54.835318    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2t57w"
Nov 03 17:45:54.839846 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:54.839826    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f85t9\""
Nov 03 17:45:54.842265 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:54.842234    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xjw2"
Nov 03 17:45:54.849921 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:54.849704    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z264t"
Nov 03 17:45:54.905138 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:54.905107    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-dwxs2" event={"ID":"e106ef93-623a-4cb4-9848-31087f1a7711","Type":"ContainerStarted","Data":"811729dc2b0bedcc93b06f87c2607de244970646668de140420a0a00d3c069fe"}
Nov 03 17:45:54.905976 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:54.905916    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-dwxs2"
Nov 03 17:45:54.912332 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:54.912309    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-dwxs2"
Nov 03 17:45:55.142923 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:55.142817    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-dwxs2" podStartSLOduration=1.826780254 podStartE2EDuration="6.142800913s" totalImagesPullingTime="4.316020659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:45:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:55.023534476 +0000 UTC m=+65.999182231" watchObservedRunningTime="2026-11-03 17:45:55.142800913 +0000 UTC m=+66.118448691"
Nov 03 17:45:55.217471 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:55.217450    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2t57w"]
Nov 03 17:45:55.219437 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:45:55.219403    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43c9da28_334d_4bb3_a6bc_e3da2d561633.slice/crio-548bb6c5c09ad0038c341aa2088a3cd11b1911863d21f19dc91b1d728a093136 WatchSource:0}: Error finding container 548bb6c5c09ad0038c341aa2088a3cd11b1911863d21f19dc91b1d728a093136: Status 404 returned error can't find the container with id 548bb6c5c09ad0038c341aa2088a3cd11b1911863d21f19dc91b1d728a093136
Nov 03 17:45:55.267751 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:55.267725    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4xjw2"]
Nov 03 17:45:55.270212 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:55.270191    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-z264t"]
Nov 03 17:45:55.270264 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:45:55.270196    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4e4b7a9_59f2_492e_a1d1_7d97a30cdc30.slice/crio-2d40add9b19ed573199dbc143ab57a05279318c4f443a5f8ccd193dbed28b065 WatchSource:0}: Error finding container 2d40add9b19ed573199dbc143ab57a05279318c4f443a5f8ccd193dbed28b065: Status 404 returned error can't find the container with id 2d40add9b19ed573199dbc143ab57a05279318c4f443a5f8ccd193dbed28b065
Nov 03 17:45:55.272053 ip-10-0-128-204 kubenswrapper[2576]: W1103 17:45:55.272030    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cf65c17_1c79_4cd5_9ef1_dceb467878b0.slice/crio-92784a8e82349f46b51d0d81cfa3ba97b2d371b63d606eaf1c457b088f74c2cd WatchSource:0}: Error finding container 92784a8e82349f46b51d0d81cfa3ba97b2d371b63d606eaf1c457b088f74c2cd: Status 404 returned error can't find the container with id 92784a8e82349f46b51d0d81cfa3ba97b2d371b63d606eaf1c457b088f74c2cd
Nov 03 17:45:55.909545 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:55.909511    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2t57w" event={"ID":"43c9da28-334d-4bb3-a6bc-e3da2d561633","Type":"ContainerStarted","Data":"548bb6c5c09ad0038c341aa2088a3cd11b1911863d21f19dc91b1d728a093136"}
Nov 03 17:45:55.910657 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:55.910635    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z264t" event={"ID":"9cf65c17-1c79-4cd5-9ef1-dceb467878b0","Type":"ContainerStarted","Data":"92784a8e82349f46b51d0d81cfa3ba97b2d371b63d606eaf1c457b088f74c2cd"}
Nov 03 17:45:55.911747 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:55.911725    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4xjw2" event={"ID":"f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30","Type":"ContainerStarted","Data":"2d40add9b19ed573199dbc143ab57a05279318c4f443a5f8ccd193dbed28b065"}
Nov 03 17:45:57.924509 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:57.924403    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z264t" event={"ID":"9cf65c17-1c79-4cd5-9ef1-dceb467878b0","Type":"ContainerStarted","Data":"b020dad57b253e69a45decda9e00505f9bd81c96f24ea6135e96e6ab2f7e6795"}
Nov 03 17:45:57.924509 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:57.924472    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z264t" event={"ID":"9cf65c17-1c79-4cd5-9ef1-dceb467878b0","Type":"ContainerStarted","Data":"6f7bb578a8011ff3ea0e07279be00c817f75c92b8569bc65632db0fe7318b494"}
Nov 03 17:45:58.045310 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:58.045249    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-z264t" podStartSLOduration=67.313331146 podStartE2EDuration="1m9.045231006s" totalImagesPullingTime="1.73189986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:58.044067278 +0000 UTC m=+69.019715022" watchObservedRunningTime="2026-11-03 17:45:58.045231006 +0000 UTC m=+69.020878750"
Nov 03 17:45:59.913103 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:59.913070    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr"
Nov 03 17:45:59.933698 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:59.933656    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2t57w" event={"ID":"43c9da28-334d-4bb3-a6bc-e3da2d561633","Type":"ContainerStarted","Data":"adbc00082e4c3ee61fdc54ec8aace91559730b8fe216b6a982e37494021cf5ec"}
Nov 03 17:45:59.933866 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:45:59.933752    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2t57w"
Nov 03 17:46:00.236279 ip-10-0-128-204 kubenswrapper[2576]: I1103 17:46:00.236167    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2t57w" podStartSLOduration=67.351371278 podStartE2EDuration="1m11.236146087s" totalImagesPullingTime="3.884774809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:46:00.234537368 +0000 UTC m=+71.210185112" watchObservedRunningTime="2026-11-03 17:46:00.236146087 +0000 UTC m=+71.211793836"
Nov 03 17:46:00.909122 ip-10-0-128-204 systemd[1]: Stopping Kubernetes Kubelet...
Nov 03 17:46:00.921011 ip-10-0-128-204 systemd[1]: kubelet.service: Deactivated successfully.
Nov 03 17:46:00.921179 ip-10-0-128-204 systemd[1]: Stopped Kubernetes Kubelet.
Nov 03 17:46:00.921228 ip-10-0-128-204 systemd[1]: kubelet.service: Consumed 3.856s CPU time.
Nov 03 17:46:00.936393 ip-10-0-128-204 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 17:46:01.014861 ip-10-0-128-204 kubenswrapper[5315]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 17:46:01.014861 ip-10-0-128-204 kubenswrapper[5315]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 17:46:01.014861 ip-10-0-128-204 kubenswrapper[5315]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 17:46:01.014861 ip-10-0-128-204 kubenswrapper[5315]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 17:46:01.016952 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.016853    5315 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:46:01.016952 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.016952    5315 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:46:01.017012 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.016957    5315 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:46:01.017012 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.016960    5315 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:46:01.017012 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.016964    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:46:01.017012 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.016967    5315 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:46:01.017012 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.016970    5315 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:46:01.017012 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.016972    5315 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:46:01.017012 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.016975    5315 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:46:01.017012 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.016978    5315 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:46:01.017012 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.016981    5315 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:46:01.017012 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.016983    5315 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:46:01.017012 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.016986    5315 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:46:01.017012 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.016989    5315 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:46:01.017012 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.016992    5315 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:46:01.017012 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.016995    5315 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:46:01.017012 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.016998    5315 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:46:01.017012 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017000    5315 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:46:01.017012 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017003    5315 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:46:01.017012 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017006    5315 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:46:01.017012 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017009    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:46:01.017479 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017011    5315 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:46:01.017479 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017017    5315 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:46:01.017479 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017020    5315 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:46:01.017479 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017022    5315 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:46:01.017479 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017025    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:46:01.017479 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017033    5315 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:46:01.017479 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017036    5315 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:46:01.017479 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017039    5315 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:46:01.017479 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017042    5315 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:46:01.017479 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017044    5315 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:46:01.017479 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017047    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:46:01.017479 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017049    5315 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:46:01.017479 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017052    5315 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:46:01.017479 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017055    5315 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:46:01.017479 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017057    5315 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:46:01.017479 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017060    5315 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:46:01.017479 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017062    5315 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:46:01.017479 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017065    5315 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:46:01.017479 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017068    5315 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:46:01.017479 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017070    5315 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:46:01.017983 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017073    5315 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:46:01.017983 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017075    5315 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:46:01.017983 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017078    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:46:01.017983 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017080    5315 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:46:01.017983 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017084    5315 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:46:01.017983 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017088    5315 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:46:01.017983 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017091    5315 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:46:01.017983 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017093    5315 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:46:01.017983 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017096    5315 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:46:01.017983 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017099    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:46:01.017983 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017101    5315 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:46:01.017983 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017104    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:46:01.017983 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017106    5315 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:46:01.017983 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017111    5315 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:46:01.017983 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017113    5315 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:46:01.017983 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017117    5315 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:46:01.017983 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017121    5315 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:46:01.017983 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017125    5315 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:46:01.017983 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017128    5315 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:46:01.018431 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017131    5315 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:46:01.018431 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017134    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:46:01.018431 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017137    5315 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:46:01.018431 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017139    5315 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:46:01.018431 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017142    5315 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:46:01.018431 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017145    5315 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:46:01.018431 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017147    5315 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:46:01.018431 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017150    5315 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:46:01.018431 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017152    5315 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:46:01.018431 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017155    5315 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:46:01.018431 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017157    5315 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:46:01.018431 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017160    5315 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:46:01.018431 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017162    5315 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:46:01.018431 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017165    5315 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:46:01.018431 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017169    5315 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:46:01.018431 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017171    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:46:01.018431 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017174    5315 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:46:01.018431 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017176    5315 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:46:01.018431 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017179    5315 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:46:01.018431 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017182    5315 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:46:01.018934 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017184    5315 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:46:01.018934 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017187    5315 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:46:01.018934 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017190    5315 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:46:01.018934 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017192    5315 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:46:01.018934 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017195    5315 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:46:01.018934 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017198    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:46:01.018934 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017202    5315 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:46:01.018934 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017204    5315 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:46:01.018934 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017207    5315 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:46:01.018934 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017210    5315 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:46:01.018934 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017213    5315 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:46:01.018934 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017215    5315 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:46:01.018934 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017218    5315 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:46:01.018934 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017221    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:46:01.018934 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017224    5315 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:46:01.018934 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017226    5315 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:46:01.018934 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017229    5315 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:46:01.018934 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017231    5315 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:46:01.018934 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017234    5315 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:46:01.018934 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017237    5315 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:46:01.019481 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017239    5315 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:46:01.019481 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017242    5315 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:46:01.019481 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017244    5315 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:46:01.019481 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017248    5315 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:46:01.019481 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017251    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:46:01.019481 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.017253    5315 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:46:01.019481 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018014    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:46:01.019481 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018024    5315 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:46:01.019481 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018029    5315 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:46:01.019481 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018033    5315 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:46:01.019481 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018037    5315 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:46:01.019481 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018041    5315 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:46:01.019481 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018044    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:46:01.019481 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018047    5315 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:46:01.019481 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018050    5315 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:46:01.019481 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018053    5315 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:46:01.019481 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018055    5315 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:46:01.019481 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018058    5315 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:46:01.019481 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018062    5315 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:46:01.019481 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018064    5315 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:46:01.019991 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018067    5315 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:46:01.019991 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018069    5315 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:46:01.019991 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018072    5315 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:46:01.019991 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018075    5315 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:46:01.019991 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018077    5315 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:46:01.019991 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018080    5315 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:46:01.019991 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018083    5315 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:46:01.019991 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018086    5315 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:46:01.019991 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018088    5315 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:46:01.019991 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018091    5315 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:46:01.019991 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018093    5315 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:46:01.019991 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018096    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:46:01.019991 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018099    5315 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:46:01.019991 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018101    5315 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:46:01.019991 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018105    5315 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:46:01.019991 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018109    5315 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:46:01.019991 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018112    5315 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:46:01.019991 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018115    5315 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:46:01.019991 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018118    5315 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:46:01.020488 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018121    5315 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:46:01.020488 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018123    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:46:01.020488 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018126    5315 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:46:01.020488 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018129    5315 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:46:01.020488 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018131    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:46:01.020488 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018134    5315 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:46:01.020488 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018137    5315 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:46:01.020488 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018139    5315 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:46:01.020488 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018142    5315 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:46:01.020488 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018145    5315 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:46:01.020488 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018147    5315 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:46:01.020488 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018150    5315 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:46:01.020488 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018154    5315 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:46:01.020488 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018156    5315 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:46:01.020488 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018159    5315 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:46:01.020488 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018161    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:46:01.020488 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018164    5315 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:46:01.020488 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018166    5315 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:46:01.020488 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018169    5315 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:46:01.020488 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018176    5315 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:46:01.021006 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018179    5315 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:46:01.021006 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018182    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:46:01.021006 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018185    5315 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:46:01.021006 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018188    5315 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:46:01.021006 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018190    5315 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:46:01.021006 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018193    5315 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:46:01.021006 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018196    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:46:01.021006 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018198    5315 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:46:01.021006 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018201    5315 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:46:01.021006 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018203    5315 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:46:01.021006 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018206    5315 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:46:01.021006 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018208    5315 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:46:01.021006 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018211    5315 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:46:01.021006 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018214    5315 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:46:01.021006 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018216    5315 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:46:01.021006 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018219    5315 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:46:01.021006 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018221    5315 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:46:01.021006 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018224    5315 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:46:01.021006 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018229    5315 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:46:01.021495 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018231    5315 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:46:01.021495 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018234    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:46:01.021495 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018236    5315 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:46:01.021495 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018239    5315 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:46:01.021495 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018242    5315 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:46:01.021495 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018245    5315 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:46:01.021495 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018247    5315 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:46:01.021495 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018250    5315 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:46:01.021495 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018252    5315 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:46:01.021495 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018255    5315 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:46:01.021495 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018257    5315 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:46:01.021495 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018260    5315 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:46:01.021495 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018262    5315 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:46:01.021495 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018266    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:46:01.021495 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018269    5315 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:46:01.021495 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018272    5315 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:46:01.021495 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018274    5315 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:46:01.021495 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018277    5315 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:46:01.021495 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018279    5315 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:46:01.021495 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018282    5315 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:46:01.021992 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018285    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:46:01.021992 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018287    5315 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:46:01.021992 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018290    5315 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:46:01.021992 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018292    5315 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:46:01.021992 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018295    5315 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:46:01.021992 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018298    5315 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:46:01.021992 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018300    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:46:01.021992 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018303    5315 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:46:01.021992 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018305    5315 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:46:01.021992 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018308    5315 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:46:01.021992 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018310    5315 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:46:01.021992 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018314    5315 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:46:01.021992 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018318    5315 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:46:01.021992 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018321    5315 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:46:01.021992 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018857    5315 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:46:01.021992 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018863    5315 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:46:01.021992 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018866    5315 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:46:01.021992 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018869    5315 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:46:01.021992 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018873    5315 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:46:01.022443 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018876    5315 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:46:01.022443 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018879    5315 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:46:01.022443 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018882    5315 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:46:01.022443 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018884    5315 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:46:01.022443 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018887    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:46:01.022443 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018889    5315 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:46:01.022443 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018892    5315 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:46:01.022443 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018895    5315 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:46:01.022443 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018898    5315 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:46:01.022443 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018903    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:46:01.022443 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018906    5315 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:46:01.022443 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018910    5315 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:46:01.022443 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018914    5315 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:46:01.022443 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018918    5315 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:46:01.022443 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018920    5315 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:46:01.022443 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018923    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:46:01.022443 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018925    5315 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:46:01.022443 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018928    5315 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:46:01.022879 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018931    5315 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:46:01.022879 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018933    5315 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:46:01.022879 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018936    5315 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:46:01.022879 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018940    5315 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:46:01.022879 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018945    5315 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:46:01.022879 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018949    5315 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:46:01.022879 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018955    5315 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:46:01.022879 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018959    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:46:01.022879 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018963    5315 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:46:01.022879 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018967    5315 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:46:01.022879 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018971    5315 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:46:01.022879 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018975    5315 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:46:01.022879 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018980    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:46:01.022879 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018982    5315 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:46:01.022879 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018985    5315 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:46:01.022879 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018988    5315 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:46:01.022879 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018990    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:46:01.022879 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018993    5315 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:46:01.022879 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018996    5315 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:46:01.023671 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.018999    5315 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:46:01.023671 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019001    5315 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:46:01.023671 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019006    5315 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:46:01.023671 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019009    5315 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:46:01.023671 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019015    5315 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:46:01.023671 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019019    5315 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:46:01.023671 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019023    5315 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:46:01.023671 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019027    5315 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:46:01.023671 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019030    5315 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:46:01.023671 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019033    5315 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:46:01.023671 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019036    5315 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:46:01.023671 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019038    5315 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:46:01.023671 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019041    5315 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:46:01.023671 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019044    5315 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:46:01.023671 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019048    5315 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:46:01.023671 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019051    5315 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:46:01.023671 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019055    5315 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:46:01.023671 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019062    5315 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:46:01.023671 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019067    5315 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:46:01.023671 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019073    5315 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:46:01.024556 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019087    5315 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:46:01.024556 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019092    5315 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:46:01.024556 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019096    5315 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:46:01.024556 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019100    5315 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:46:01.024556 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019103    5315 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:46:01.024556 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019107    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:46:01.024556 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019111    5315 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:46:01.024556 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019115    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:46:01.024556 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019119    5315 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:46:01.024556 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019123    5315 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:46:01.024556 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019126    5315 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:46:01.024556 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019130    5315 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:46:01.024556 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019134    5315 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:46:01.024556 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019138    5315 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:46:01.024556 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019142    5315 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:46:01.024556 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019158    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:46:01.024556 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019163    5315 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:46:01.024556 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019167    5315 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:46:01.024556 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019171    5315 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:46:01.024556 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019175    5315 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:46:01.025427 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019179    5315 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:46:01.025427 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019183    5315 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:46:01.025427 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019188    5315 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:46:01.025427 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019192    5315 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:46:01.025427 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019196    5315 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:46:01.025427 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019200    5315 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:46:01.025427 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019203    5315 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:46:01.025427 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019207    5315 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:46:01.025427 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019211    5315 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:46:01.025427 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019215    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:46:01.025427 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019219    5315 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:46:01.025427 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019225    5315 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:46:01.025427 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019228    5315 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:46:01.025427 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019232    5315 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:46:01.025427 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019235    5315 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:46:01.025427 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019238    5315 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:46:01.025427 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019240    5315 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:46:01.025427 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019243    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:46:01.025427 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019246    5315 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:46:01.025427 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019250    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:46:01.026361 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019253    5315 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:46:01.026361 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019255    5315 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:46:01.026361 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019258    5315 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:46:01.026361 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.019260    5315 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:46:01.026361 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019361    5315 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 17:46:01.026361 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019368    5315 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 17:46:01.026361 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019381    5315 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 17:46:01.026361 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019385    5315 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 17:46:01.026361 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019393    5315 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 17:46:01.026361 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019396    5315 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 17:46:01.026361 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019401    5315 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 17:46:01.026361 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019405    5315 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 17:46:01.026361 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019409    5315 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 17:46:01.026361 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019412    5315 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 17:46:01.026361 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019415    5315 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 17:46:01.026361 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019419    5315 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 17:46:01.026361 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019422    5315 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 17:46:01.026361 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019425    5315 flags.go:64] FLAG: --cgroup-root=""
Nov 03 17:46:01.026361 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019428    5315 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 17:46:01.026361 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019431    5315 flags.go:64] FLAG: --client-ca-file=""
Nov 03 17:46:01.026361 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019434    5315 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 17:46:01.026361 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019437    5315 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 17:46:01.026361 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019444    5315 flags.go:64] FLAG: --cluster-domain=""
Nov 03 17:46:01.026361 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019448    5315 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 17:46:01.027249 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019451    5315 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 17:46:01.027249 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019455    5315 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 17:46:01.027249 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019458    5315 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 17:46:01.027249 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019462    5315 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 17:46:01.027249 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019465    5315 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 17:46:01.027249 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019468    5315 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 17:46:01.027249 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019472    5315 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 17:46:01.027249 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019475    5315 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 17:46:01.027249 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019478    5315 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 17:46:01.027249 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019481    5315 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 17:46:01.027249 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019484    5315 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 17:46:01.027249 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019487    5315 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 17:46:01.027249 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019499    5315 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 17:46:01.027249 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019502    5315 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 17:46:01.027249 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019505    5315 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 17:46:01.027249 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019508    5315 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 17:46:01.027249 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019512    5315 flags.go:64] FLAG: --enable-server="true"
Nov 03 17:46:01.027249 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019516    5315 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 17:46:01.027249 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019521    5315 flags.go:64] FLAG: --event-burst="100"
Nov 03 17:46:01.027249 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019524    5315 flags.go:64] FLAG: --event-qps="50"
Nov 03 17:46:01.027249 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019527    5315 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 17:46:01.027249 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019530    5315 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 17:46:01.027249 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019533    5315 flags.go:64] FLAG: --eviction-hard=""
Nov 03 17:46:01.027249 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019541    5315 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 17:46:01.027877 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019544    5315 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 17:46:01.027877 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019547    5315 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 17:46:01.027877 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019550    5315 flags.go:64] FLAG: --eviction-soft=""
Nov 03 17:46:01.027877 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019558    5315 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 17:46:01.027877 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019562    5315 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 17:46:01.027877 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019565    5315 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 17:46:01.027877 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019568    5315 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 17:46:01.027877 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019571    5315 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 17:46:01.027877 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019574    5315 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 17:46:01.027877 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019577    5315 flags.go:64] FLAG: --feature-gates=""
Nov 03 17:46:01.027877 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019582    5315 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 17:46:01.027877 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019585    5315 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 17:46:01.027877 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019588    5315 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 17:46:01.027877 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019591    5315 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 17:46:01.027877 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019594    5315 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 17:46:01.027877 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019597    5315 flags.go:64] FLAG: --help="false"
Nov 03 17:46:01.027877 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019600    5315 flags.go:64] FLAG: --hostname-override="ip-10-0-128-204.us-west-2.compute.internal"
Nov 03 17:46:01.027877 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019604    5315 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 17:46:01.027877 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019619    5315 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 17:46:01.027877 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019622    5315 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 17:46:01.027877 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019626    5315 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 17:46:01.027877 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019629    5315 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 17:46:01.027877 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019633    5315 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 17:46:01.028462 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019636    5315 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 17:46:01.028462 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019640    5315 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 17:46:01.028462 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019643    5315 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 17:46:01.028462 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019646    5315 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 17:46:01.028462 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019650    5315 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 17:46:01.028462 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019653    5315 flags.go:64] FLAG: --kube-reserved=""
Nov 03 17:46:01.028462 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019655    5315 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 17:46:01.028462 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019658    5315 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 17:46:01.028462 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019661    5315 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 17:46:01.028462 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019664    5315 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 17:46:01.028462 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019668    5315 flags.go:64] FLAG: --lock-file=""
Nov 03 17:46:01.028462 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019671    5315 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 17:46:01.028462 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019674    5315 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 17:46:01.028462 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019677    5315 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 17:46:01.028462 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019682    5315 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 17:46:01.028462 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019685    5315 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 17:46:01.028462 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019689    5315 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 17:46:01.028462 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019692    5315 flags.go:64] FLAG: --logging-format="text"
Nov 03 17:46:01.028462 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019695    5315 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 17:46:01.028462 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019698    5315 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 17:46:01.028462 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019701    5315 flags.go:64] FLAG: --manifest-url=""
Nov 03 17:46:01.028462 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019704    5315 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 17:46:01.028462 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019709    5315 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 17:46:01.028462 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019712    5315 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 17:46:01.028462 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019716    5315 flags.go:64] FLAG: --max-pods="110"
Nov 03 17:46:01.029083 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019719    5315 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 17:46:01.029083 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019721    5315 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 17:46:01.029083 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019725    5315 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 17:46:01.029083 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019728    5315 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 17:46:01.029083 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019730    5315 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 17:46:01.029083 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019733    5315 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 17:46:01.029083 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019736    5315 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 17:46:01.029083 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019743    5315 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 17:46:01.029083 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019749    5315 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 17:46:01.029083 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019752    5315 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 17:46:01.029083 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019755    5315 flags.go:64] FLAG: --pod-cidr=""
Nov 03 17:46:01.029083 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019764    5315 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 17:46:01.029083 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019767    5315 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 17:46:01.029083 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019770    5315 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 17:46:01.029083 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019773    5315 flags.go:64] FLAG: --port="10250"
Nov 03 17:46:01.029083 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019776    5315 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 17:46:01.029083 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019779    5315 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0e294fd8eca1f506e"
Nov 03 17:46:01.029083 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019782    5315 flags.go:64] FLAG: --qos-reserved=""
Nov 03 17:46:01.029083 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019785    5315 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 17:46:01.029083 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019788    5315 flags.go:64] FLAG: --register-node="true"
Nov 03 17:46:01.029083 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019791    5315 flags.go:64] FLAG: --register-with-taints=""
Nov 03 17:46:01.029083 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019795    5315 flags.go:64] FLAG: --registry-burst="10"
Nov 03 17:46:01.029083 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019798    5315 flags.go:64] FLAG: --registry-qps="5"
Nov 03 17:46:01.029083 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019801    5315 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 17:46:01.029083 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019804    5315 flags.go:64] FLAG: --reserved-memory=""
Nov 03 17:46:01.029083 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019808    5315 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 17:46:01.029714 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019811    5315 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 17:46:01.029714 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019814    5315 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 17:46:01.029714 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019816    5315 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 17:46:01.029714 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019819    5315 flags.go:64] FLAG: --runonce="false"
Nov 03 17:46:01.029714 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019822    5315 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 17:46:01.029714 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019826    5315 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 17:46:01.029714 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019829    5315 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 17:46:01.029714 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019832    5315 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 17:46:01.029714 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019835    5315 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 17:46:01.029714 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019838    5315 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 17:46:01.029714 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019841    5315 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 17:46:01.029714 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019844    5315 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 17:46:01.029714 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019847    5315 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 17:46:01.029714 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019850    5315 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 17:46:01.029714 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019855    5315 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 17:46:01.029714 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019857    5315 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 17:46:01.029714 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019860    5315 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 17:46:01.029714 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019863    5315 flags.go:64] FLAG: --system-cgroups=""
Nov 03 17:46:01.029714 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019872    5315 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 17:46:01.029714 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019877    5315 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 17:46:01.029714 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019880    5315 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 17:46:01.029714 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019883    5315 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 17:46:01.029714 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019889    5315 flags.go:64] FLAG: --tls-min-version=""
Nov 03 17:46:01.029714 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019894    5315 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 17:46:01.029714 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019898    5315 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 17:46:01.030445 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019902    5315 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 17:46:01.030445 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019906    5315 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 17:46:01.030445 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019910    5315 flags.go:64] FLAG: --v="2"
Nov 03 17:46:01.030445 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019917    5315 flags.go:64] FLAG: --version="false"
Nov 03 17:46:01.030445 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019922    5315 flags.go:64] FLAG: --vmodule=""
Nov 03 17:46:01.030445 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019926    5315 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 17:46:01.030445 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.019929    5315 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 17:46:01.030445 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020153    5315 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:46:01.030445 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020160    5315 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:46:01.030445 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020164    5315 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:46:01.030445 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020169    5315 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:46:01.030445 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020173    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:46:01.030445 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020176    5315 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:46:01.030445 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020180    5315 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:46:01.030445 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020183    5315 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:46:01.030445 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020186    5315 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:46:01.030445 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020189    5315 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:46:01.030445 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020192    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:46:01.030445 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020194    5315 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:46:01.030445 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020197    5315 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:46:01.030445 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020200    5315 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:46:01.030995 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020205    5315 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:46:01.030995 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020207    5315 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:46:01.030995 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020212    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:46:01.030995 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020214    5315 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:46:01.030995 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020218    5315 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:46:01.030995 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020226    5315 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:46:01.030995 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020229    5315 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:46:01.030995 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020239    5315 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:46:01.030995 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020241    5315 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:46:01.030995 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020244    5315 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:46:01.030995 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020247    5315 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:46:01.030995 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020250    5315 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:46:01.030995 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020252    5315 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:46:01.030995 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020255    5315 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:46:01.030995 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020258    5315 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:46:01.030995 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020261    5315 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:46:01.030995 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020263    5315 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:46:01.030995 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020266    5315 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:46:01.030995 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020268    5315 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:46:01.030995 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020271    5315 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:46:01.031461 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020274    5315 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:46:01.031461 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020277    5315 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:46:01.031461 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020279    5315 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:46:01.031461 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020282    5315 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:46:01.031461 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020285    5315 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:46:01.031461 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020297    5315 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:46:01.031461 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020300    5315 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:46:01.031461 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020303    5315 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:46:01.031461 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020306    5315 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:46:01.031461 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020308    5315 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:46:01.031461 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020311    5315 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:46:01.031461 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020314    5315 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:46:01.031461 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020318    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:46:01.031461 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020321    5315 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:46:01.031461 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020325    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:46:01.031461 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020327    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:46:01.031461 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020330    5315 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:46:01.031461 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020333    5315 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:46:01.031461 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020342    5315 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:46:01.031461 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020345    5315 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:46:01.032133 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020347    5315 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:46:01.032133 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020350    5315 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:46:01.032133 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020353    5315 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:46:01.032133 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020355    5315 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:46:01.032133 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020358    5315 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:46:01.032133 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020361    5315 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:46:01.032133 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020364    5315 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:46:01.032133 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020366    5315 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:46:01.032133 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020369    5315 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:46:01.032133 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020372    5315 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:46:01.032133 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020374    5315 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:46:01.032133 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020377    5315 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:46:01.032133 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020380    5315 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:46:01.032133 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020382    5315 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:46:01.032133 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020385    5315 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:46:01.032133 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020387    5315 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:46:01.032133 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020390    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:46:01.032133 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020393    5315 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:46:01.032133 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020396    5315 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:46:01.032713 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020398    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:46:01.032713 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020401    5315 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:46:01.032713 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020403    5315 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:46:01.032713 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020406    5315 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:46:01.032713 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020409    5315 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:46:01.032713 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020413    5315 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:46:01.032713 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020415    5315 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:46:01.032713 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020419    5315 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:46:01.032713 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020422    5315 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:46:01.032713 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020425    5315 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:46:01.032713 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020427    5315 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:46:01.032713 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020430    5315 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:46:01.032713 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020439    5315 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:46:01.032713 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020443    5315 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:46:01.032713 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020445    5315 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:46:01.032713 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020448    5315 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:46:01.032713 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020450    5315 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:46:01.032713 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020453    5315 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:46:01.032713 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020456    5315 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:46:01.033359 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020458    5315 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:46:01.033359 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020461    5315 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:46:01.033359 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020463    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:46:01.033359 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020466    5315 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:46:01.033359 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020469    5315 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:46:01.033359 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020471    5315 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:46:01.033359 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020474    5315 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:46:01.033359 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020476    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:46:01.033359 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020479    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:46:01.033359 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020482    5315 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:46:01.033359 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020485    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:46:01.033359 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020487    5315 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:46:01.033359 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020490    5315 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:46:01.033359 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020493    5315 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:46:01.033359 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.020523    5315 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 17:46:01.033870 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020566    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:46:01.033870 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020571    5315 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:46:01.033870 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020574    5315 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:46:01.033870 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020578    5315 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:46:01.033870 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020582    5315 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:46:01.033870 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020586    5315 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:46:01.033870 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020589    5315 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:46:01.033870 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020592    5315 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:46:01.033870 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020594    5315 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:46:01.033870 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020597    5315 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:46:01.033870 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020600    5315 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:46:01.033870 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020619    5315 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:46:01.033870 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020622    5315 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:46:01.033870 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020625    5315 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:46:01.033870 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020628    5315 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:46:01.033870 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020630    5315 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:46:01.033870 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020633    5315 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:46:01.033870 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020636    5315 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:46:01.033870 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020638    5315 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:46:01.033870 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020641    5315 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:46:01.034367 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020644    5315 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:46:01.034367 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020646    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:46:01.034367 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020649    5315 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:46:01.034367 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020652    5315 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:46:01.034367 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020655    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:46:01.034367 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020657    5315 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:46:01.034367 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020660    5315 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:46:01.034367 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020663    5315 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:46:01.034367 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020665    5315 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:46:01.034367 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020668    5315 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:46:01.034367 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020670    5315 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:46:01.034367 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020673    5315 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:46:01.034367 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020676    5315 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:46:01.034367 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020678    5315 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:46:01.034367 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020681    5315 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:46:01.034367 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020683    5315 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:46:01.034367 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020689    5315 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:46:01.034367 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020693    5315 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:46:01.034367 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020696    5315 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:46:01.034905 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020699    5315 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:46:01.034905 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020702    5315 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:46:01.034905 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020704    5315 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:46:01.034905 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020707    5315 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:46:01.034905 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020709    5315 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:46:01.034905 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020717    5315 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:46:01.034905 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020720    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:46:01.034905 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020723    5315 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:46:01.034905 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020726    5315 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:46:01.034905 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020728    5315 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:46:01.034905 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020731    5315 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:46:01.034905 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020733    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:46:01.034905 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020736    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:46:01.034905 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020739    5315 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:46:01.034905 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020741    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:46:01.034905 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020744    5315 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:46:01.034905 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020747    5315 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:46:01.034905 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020749    5315 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:46:01.034905 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020752    5315 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:46:01.035456 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020754    5315 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:46:01.035456 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020757    5315 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:46:01.035456 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020761    5315 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:46:01.035456 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020765    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:46:01.035456 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020768    5315 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:46:01.035456 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020771    5315 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:46:01.035456 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020773    5315 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:46:01.035456 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020776    5315 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:46:01.035456 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020778    5315 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:46:01.035456 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020781    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:46:01.035456 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020785    5315 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:46:01.035456 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020789    5315 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:46:01.035456 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020792    5315 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:46:01.035456 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020794    5315 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:46:01.035456 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020797    5315 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:46:01.035456 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020799    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:46:01.035456 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020802    5315 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:46:01.035456 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020805    5315 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:46:01.035456 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020808    5315 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:46:01.035456 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020815    5315 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:46:01.036127 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020818    5315 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:46:01.036127 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020821    5315 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:46:01.036127 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020824    5315 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:46:01.036127 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020826    5315 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:46:01.036127 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020829    5315 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:46:01.036127 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020831    5315 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:46:01.036127 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020834    5315 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:46:01.036127 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020837    5315 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:46:01.036127 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020839    5315 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:46:01.036127 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020842    5315 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:46:01.036127 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020844    5315 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:46:01.036127 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020847    5315 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:46:01.036127 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020849    5315 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:46:01.036127 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020852    5315 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:46:01.036127 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020854    5315 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:46:01.036127 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020857    5315 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:46:01.036127 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020860    5315 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:46:01.036127 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020862    5315 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:46:01.036127 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020865    5315 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:46:01.036626 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020867    5315 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:46:01.036626 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020870    5315 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:46:01.036626 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020873    5315 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:46:01.036626 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020877    5315 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:46:01.036626 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020881    5315 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:46:01.036626 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020883    5315 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:46:01.036626 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020886    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:46:01.036626 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020889    5315 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:46:01.036626 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.020892    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:46:01.036626 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.026406    5315 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 17:46:01.036626 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.026443    5315 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 17:46:01.036626 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.026475    5315 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 17:46:01.036626 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.026484    5315 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 17:46:01.036626 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026700    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:46:01.036626 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026711    5315 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:46:01.036626 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026717    5315 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:46:01.036626 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026728    5315 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:46:01.036626 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026733    5315 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:46:01.036626 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026738    5315 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:46:01.037085 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026742    5315 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:46:01.037085 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026747    5315 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:46:01.037085 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026752    5315 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:46:01.037085 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026758    5315 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:46:01.037085 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026763    5315 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:46:01.037085 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026768    5315 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:46:01.037085 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026772    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:46:01.037085 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026777    5315 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:46:01.037085 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026781    5315 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:46:01.037085 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026791    5315 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:46:01.037085 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026796    5315 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:46:01.037085 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026802    5315 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:46:01.037085 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026806    5315 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:46:01.037085 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026811    5315 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:46:01.037085 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026816    5315 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:46:01.037085 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026824    5315 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:46:01.037085 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026830    5315 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:46:01.037085 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026835    5315 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:46:01.037085 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026840    5315 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:46:01.037543 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026845    5315 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:46:01.037543 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026850    5315 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:46:01.037543 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026855    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:46:01.037543 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026865    5315 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:46:01.037543 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026870    5315 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:46:01.037543 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026876    5315 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:46:01.037543 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026881    5315 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:46:01.037543 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026886    5315 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:46:01.037543 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026891    5315 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:46:01.037543 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026895    5315 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:46:01.037543 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026900    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:46:01.037543 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026905    5315 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:46:01.037543 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026910    5315 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:46:01.037543 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026914    5315 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:46:01.037543 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026919    5315 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:46:01.037543 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026924    5315 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:46:01.037543 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026933    5315 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:46:01.037543 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026939    5315 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:46:01.037543 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026944    5315 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:46:01.037543 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026949    5315 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:46:01.038091 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026953    5315 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:46:01.038091 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026958    5315 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:46:01.038091 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026962    5315 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:46:01.038091 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026967    5315 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:46:01.038091 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026972    5315 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:46:01.038091 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026977    5315 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:46:01.038091 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026981    5315 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:46:01.038091 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.026986    5315 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:46:01.038091 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027460    5315 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:46:01.038091 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027584    5315 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:46:01.038091 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027590    5315 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:46:01.038091 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027596    5315 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:46:01.038091 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027600    5315 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:46:01.038091 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027604    5315 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:46:01.038091 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027622    5315 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:46:01.038091 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027627    5315 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:46:01.038091 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027632    5315 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:46:01.038091 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027636    5315 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:46:01.038091 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027640    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:46:01.038091 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027645    5315 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:46:01.038561 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027650    5315 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:46:01.038561 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027654    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:46:01.038561 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027659    5315 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:46:01.038561 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027663    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:46:01.038561 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027668    5315 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:46:01.038561 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027672    5315 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:46:01.038561 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027677    5315 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:46:01.038561 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027683    5315 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:46:01.038561 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027687    5315 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:46:01.038561 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027692    5315 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:46:01.038561 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027696    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:46:01.038561 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027703    5315 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:46:01.038561 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027710    5315 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:46:01.038561 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027715    5315 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:46:01.038561 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027720    5315 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:46:01.038561 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027725    5315 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:46:01.038561 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027731    5315 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:46:01.038561 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027736    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:46:01.038561 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027740    5315 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:46:01.039019 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027745    5315 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:46:01.039019 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027749    5315 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:46:01.039019 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027753    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:46:01.039019 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027758    5315 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:46:01.039019 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027762    5315 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:46:01.039019 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027766    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:46:01.039019 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027770    5315 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:46:01.039019 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027775    5315 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:46:01.039019 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027779    5315 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:46:01.039019 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027783    5315 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:46:01.039019 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027787    5315 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:46:01.039019 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027792    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:46:01.039019 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027796    5315 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:46:01.039019 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027801    5315 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:46:01.039019 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027804    5315 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:46:01.039019 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027809    5315 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:46:01.039019 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027813    5315 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:46:01.039019 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027820    5315 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:46:01.039019 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027825    5315 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:46:01.039467 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027831    5315 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:46:01.039467 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027838    5315 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:46:01.039467 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.027843    5315 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:46:01.039467 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.027879    5315 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 17:46:01.039467 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028031    5315 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:46:01.039467 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028037    5315 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:46:01.039467 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028042    5315 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:46:01.039467 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028045    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:46:01.039467 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028049    5315 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:46:01.039467 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028052    5315 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:46:01.039467 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028055    5315 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:46:01.039467 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028058    5315 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:46:01.039467 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028061    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:46:01.039467 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028064    5315 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:46:01.039467 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028067    5315 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:46:01.039467 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028070    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:46:01.039873 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028073    5315 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:46:01.039873 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028075    5315 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:46:01.039873 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028078    5315 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:46:01.039873 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028081    5315 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:46:01.039873 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028084    5315 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:46:01.039873 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028086    5315 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:46:01.039873 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028089    5315 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:46:01.039873 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028092    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:46:01.039873 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028094    5315 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:46:01.039873 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028097    5315 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:46:01.039873 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028099    5315 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:46:01.039873 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028102    5315 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:46:01.039873 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028104    5315 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:46:01.039873 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028107    5315 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:46:01.039873 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028109    5315 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:46:01.039873 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028112    5315 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:46:01.039873 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028115    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:46:01.039873 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028118    5315 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:46:01.039873 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028120    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:46:01.040430 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028123    5315 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:46:01.040430 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028126    5315 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:46:01.040430 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028129    5315 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:46:01.040430 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028133    5315 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:46:01.040430 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028137    5315 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:46:01.040430 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028140    5315 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:46:01.040430 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028144    5315 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:46:01.040430 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028148    5315 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:46:01.040430 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028151    5315 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:46:01.040430 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028154    5315 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:46:01.040430 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028156    5315 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:46:01.040430 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028159    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:46:01.040430 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028162    5315 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:46:01.040430 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028164    5315 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:46:01.040430 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028167    5315 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:46:01.040430 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028170    5315 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:46:01.040430 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028172    5315 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:46:01.040430 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028175    5315 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:46:01.040430 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028177    5315 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:46:01.041061 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028180    5315 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:46:01.041061 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028183    5315 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:46:01.041061 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028186    5315 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:46:01.041061 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028188    5315 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:46:01.041061 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028191    5315 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:46:01.041061 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028193    5315 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:46:01.041061 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028196    5315 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:46:01.041061 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028198    5315 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:46:01.041061 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028201    5315 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:46:01.041061 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028204    5315 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:46:01.041061 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028206    5315 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:46:01.041061 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028209    5315 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:46:01.041061 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028212    5315 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:46:01.041061 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028214    5315 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:46:01.041061 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028217    5315 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:46:01.041061 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028219    5315 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:46:01.041061 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028222    5315 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:46:01.041061 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028224    5315 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:46:01.041061 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028227    5315 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:46:01.041061 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028230    5315 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:46:01.041690 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028233    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:46:01.041690 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028236    5315 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:46:01.041690 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028239    5315 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:46:01.041690 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028242    5315 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:46:01.041690 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028245    5315 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:46:01.041690 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028247    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:46:01.041690 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028250    5315 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:46:01.041690 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028252    5315 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:46:01.041690 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028255    5315 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:46:01.041690 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028258    5315 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:46:01.041690 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028260    5315 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:46:01.041690 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028263    5315 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:46:01.041690 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028265    5315 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:46:01.041690 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028268    5315 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:46:01.041690 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028270    5315 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:46:01.041690 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028273    5315 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:46:01.041690 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028276    5315 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:46:01.041690 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028278    5315 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:46:01.041690 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028281    5315 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:46:01.041690 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028283    5315 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:46:01.042368 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028286    5315 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:46:01.042368 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028290    5315 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:46:01.042368 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028293    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:46:01.042368 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028296    5315 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:46:01.042368 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028300    5315 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:46:01.042368 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028303    5315 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:46:01.042368 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028306    5315 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:46:01.042368 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028308    5315 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:46:01.042368 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028311    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:46:01.042368 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028314    5315 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:46:01.042368 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028317    5315 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:46:01.042368 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028320    5315 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:46:01.042368 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028323    5315 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:46:01.042368 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028326    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:46:01.042368 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028329    5315 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:46:01.042368 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028332    5315 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:46:01.043008 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.028355    5315 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 17:46:01.043008 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028386    5315 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:46:01.043008 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028390    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:46:01.043008 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028394    5315 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:46:01.043008 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028397    5315 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:46:01.043008 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028400    5315 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:46:01.043008 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028402    5315 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:46:01.043008 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028405    5315 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:46:01.043008 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028408    5315 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:46:01.043008 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028410    5315 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:46:01.043008 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028413    5315 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:46:01.043008 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028415    5315 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:46:01.043008 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028418    5315 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:46:01.043008 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028421    5315 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:46:01.043008 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028423    5315 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:46:01.043008 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028426    5315 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:46:01.043642 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028428    5315 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:46:01.043642 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028431    5315 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:46:01.043642 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028435    5315 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:46:01.043642 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028439    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:46:01.043642 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028443    5315 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:46:01.043642 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028446    5315 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:46:01.043642 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028449    5315 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:46:01.043642 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028452    5315 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:46:01.043642 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028455    5315 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:46:01.043642 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028458    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:46:01.043642 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028461    5315 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:46:01.043642 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028464    5315 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:46:01.043642 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028467    5315 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:46:01.043642 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028469    5315 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:46:01.043642 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028472    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:46:01.043642 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028476    5315 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:46:01.043642 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028478    5315 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:46:01.043642 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028481    5315 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:46:01.043642 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028483    5315 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:46:01.044317 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028486    5315 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:46:01.044317 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028489    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:46:01.044317 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028491    5315 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:46:01.044317 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028494    5315 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:46:01.044317 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028496    5315 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:46:01.044317 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028499    5315 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:46:01.044317 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028502    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:46:01.044317 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028505    5315 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:46:01.044317 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028507    5315 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:46:01.044317 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028511    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:46:01.044317 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028513    5315 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:46:01.044317 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028516    5315 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:46:01.044317 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028518    5315 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:46:01.044317 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028521    5315 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:46:01.044317 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028524    5315 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:46:01.044317 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028526    5315 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:46:01.044317 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028529    5315 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:46:01.044317 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028531    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:46:01.044317 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028534    5315 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:46:01.044317 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028536    5315 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:46:01.044814 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028539    5315 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:46:01.044814 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028542    5315 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:46:01.044814 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028544    5315 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:46:01.044814 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028547    5315 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:46:01.044814 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028550    5315 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:46:01.044814 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028552    5315 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:46:01.044814 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028555    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:46:01.044814 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028557    5315 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:46:01.044814 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028560    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:46:01.044814 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028563    5315 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:46:01.044814 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028566    5315 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:46:01.044814 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028568    5315 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:46:01.044814 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028572    5315 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:46:01.044814 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028575    5315 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:46:01.044814 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028578    5315 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:46:01.044814 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028581    5315 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:46:01.044814 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028584    5315 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:46:01.044814 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028589    5315 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:46:01.044814 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028592    5315 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:46:01.045289 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028595    5315 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:46:01.045289 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028597    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:46:01.045289 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028600    5315 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:46:01.045289 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028603    5315 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:46:01.045289 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028620    5315 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:46:01.045289 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028624    5315 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:46:01.045289 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028627    5315 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:46:01.045289 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028630    5315 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:46:01.045289 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028633    5315 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:46:01.045289 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028635    5315 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:46:01.045289 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028638    5315 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:46:01.045289 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028641    5315 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:46:01.045289 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028644    5315 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:46:01.045289 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028646    5315 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:46:01.045289 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028649    5315 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:46:01.045289 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028651    5315 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:46:01.045289 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028654    5315 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:46:01.045289 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028657    5315 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:46:01.045289 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028660    5315 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:46:01.045289 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028662    5315 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:46:01.045821 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028665    5315 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:46:01.045821 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028668    5315 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:46:01.045821 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028671    5315 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:46:01.045821 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028674    5315 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:46:01.045821 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028677    5315 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:46:01.045821 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028680    5315 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:46:01.045821 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028682    5315 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:46:01.045821 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028685    5315 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:46:01.045821 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028688    5315 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:46:01.045821 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028690    5315 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:46:01.045821 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028695    5315 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:46:01.045821 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028698    5315 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:46:01.045821 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:46:01.028700    5315 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:46:01.045821 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.028763    5315 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 17:46:01.045821 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.029767    5315 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 03 17:46:01.045821 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.029830    5315 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 03 17:46:01.045821 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.030034    5315 server.go:1014] "Starting client certificate rotation"
Nov 03 17:46:01.045821 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.030187    5315 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 17:46:01.046564 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.030251    5315 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 17:39:49 +0000 UTC" deadline="2028-06-09 10:34:10.433767761 +0000 UTC"
Nov 03 17:46:01.046564 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.030291    5315 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14008h48m9.403479928s"
Nov 03 17:46:01.046564 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.030527    5315 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 17:46:01.046564 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.031659    5315 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 17:46:01.046564 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.032935    5315 log.go:25] "Validated CRI v1 runtime API"
Nov 03 17:46:01.046564 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.034688    5315 log.go:25] "Validated CRI v1 image API"
Nov 03 17:46:01.046564 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.035378    5315 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 17:46:01.046564 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.037178    5315 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8d4c12ac-84d9-41a3-8c88-9697de6cbccd:/dev/nvme0n1p3 d94ab8a4-0625-4ec3-a82f-ac022d9cf853:/dev/nvme0n1p4]
Nov 03 17:46:01.046941 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.037199    5315 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/00bbb7d679385391ad209f99b21a55fceb70b6ed883a692a74de60e11e9024d6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/00bbb7d679385391ad209f99b21a55fceb70b6ed883a692a74de60e11e9024d6/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0bb1fb1187d8ea80dd6c6dda38067a5c154fb2c249c4c69851fcc12aab410f85/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0bb1fb1187d8ea80dd6c6dda38067a5c154fb2c249c4c69851fcc12aab410f85/userdata/shm major:0 minor:204 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/139653ececacf7123894d53780a05a97b3a3c808ff74c54de72556fa79a00dfa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/139653ececacf7123894d53780a05a97b3a3c808ff74c54de72556fa79a00dfa/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1794a07f998f56a2861ae8bc703815fb34354f4e54a42e590f6a7535ced033d9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1794a07f998f56a2861ae8bc703815fb34354f4e54a42e590f6a7535ced033d9/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/17f5aed41c2adc3c7122f2af59303d8dd656fbdcfaba6b7b85cdd74ede8e5804/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/17f5aed41c2adc3c7122f2af59303d8dd656fbdcfaba6b7b85cdd74ede8e5804/userdata/shm major:0 minor:52 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/20eb0bd6f5eabf540587d2a2680526d0bf218614cb1ae6e6466f48b069f0dfad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/20eb0bd6f5eabf540587d2a2680526d0bf218614cb1ae6e6466f48b069f0dfad/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2d40add9b19ed573199dbc143ab57a05279318c4f443a5f8ccd193dbed28b065/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2d40add9b19ed573199dbc143ab57a05279318c4f443a5f8ccd193dbed28b065/userdata/shm major:0 minor:344 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3ad8db60b8760552cac5c6731018d7b4d80e8e8faf4944fcd6e0ec4fc4fdad1a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3ad8db60b8760552cac5c6731018d7b4d80e8e8faf4944fcd6e0ec4fc4fdad1a/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/548bb6c5c09ad0038c341aa2088a3cd11b1911863d21f19dc91b1d728a093136/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/548bb6c5c09ad0038c341aa2088a3cd11b1911863d21f19dc91b1d728a093136/userdata/shm major:0 minor:342 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/68856dcdd260964150b8ee215612e33d4f82033dbfa90a19ac4dbea63d6c3330/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/68856dcdd260964150b8ee215612e33d4f82033dbfa90a19ac4dbea63d6c3330/userdata/shm major:0 minor:298 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/735b63977e239842f058114ff665b7db5d04b38dd6ab509212b94544eaee250d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/735b63977e239842f058114ff665b7db5d04b38dd6ab509212b94544eaee250d/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7bd829a4c167033d557935b56396132797cbcee937ae350c15682fc836a324b2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7bd829a4c167033d557935b56396132797cbcee937ae350c15682fc836a324b2/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8549638c99969e115d70c745d0c81574071f2c1ac772f71c1dff07aaa52634aa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8549638c99969e115d70c745d0c81574071f2c1ac772f71c1dff07aaa52634aa/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/919fb8112573214d30fe24a1e5472e0723bcc1c57c5a2f3b98c69c3de9573523/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/919fb8112573214d30fe24a1e5472e0723bcc1c57c5a2f3b98c69c3de9573523/userdata/shm major:0 minor:271 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/92784a8e82349f46b51d0d81cfa3ba97b2d371b63d606eaf1c457b088f74c2cd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/92784a8e82349f46b51d0d81cfa3ba97b2d371b63d606eaf1c457b088f74c2cd/userdata/shm major:0 minor:346 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a493b902a49a44f6003be191f76a551ca141c08c0d1ece763fbf1d170f15e03b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a493b902a49a44f6003be191f76a551ca141c08c0d1ece763fbf1d170f15e03b/userdata/shm major:0 minor:292 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/abb3d20d17c9e1057141552b05e0c65826502f65fcb07c44d2c74c00e481b7f1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/abb3d20d17c9e1057141552b05e0c65826502f65fcb07c44d2c74c00e481b7f1/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e30ab61f1fcccd658bc913224880d4de5f884763621080cf78cb3880b3cef69f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e30ab61f1fcccd658bc913224880d4de5f884763621080cf78cb3880b3cef69f/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e6269ad31b395b408406d12e3bfbab8b2317923cdf821b7c815c69caac58878a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e6269ad31b395b408406d12e3bfbab8b2317923cdf821b7c815c69caac58878a/userdata/shm major:0 minor:73 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e9ad1fb5061bef579da494a3748707e9bc6214af69ee74945e6254c3ce22365f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e9ad1fb5061bef579da494a3748707e9bc6214af69ee74945e6254c3ce22365f/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3e881e23-d118-4245-b855-b1322d133ef1/volumes/kubernetes.io~projected/kube-api-access-9jsj9:{mountpoint:/var/lib/kubelet/pods/3e881e23-d118-4245-b855-b1322d133ef1/volumes/kubernetes.io~projected/kube-api-access-9jsj9 major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3e881e23-d118-4245-b855-b1322d133ef1/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/3e881e23-d118-4245-b855-b1322d133ef1/volumes/kubernetes.io~secret/metrics-tls major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43c9da28-334d-4bb3-a6bc-e3da2d561633/volumes/kubernetes.io~projected/kube-api-access-v89s6:{mountpoint:/var/lib/kubelet/pods/43c9da28-334d-4bb3-a6bc-e3da2d561633/volumes/kubernetes.io~projected/kube-api-access-v89s6 major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/55b54081-644c-464e-91e1-15c2d8f5be59/volumes/kubernetes.io~projected/kube-api-access-2jstr:{mountpoint:/var/lib/kubelet/pods/55b54081-644c-464e-91e1-15c2d8f5be59/volumes/kubernetes.io~projected/kube-api-access-2jstr major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/55b54081-644c-464e-91e1-15c2d8f5be59/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/55b54081-644c-464e-91e1-15c2d8f5be59/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/55b54081-644c-464e-91e1-15c2d8f5be59/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/55b54081-644c-464e-91e1-15c2d8f5be59/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e/volumes/kubernetes.io~projected/kube-api-access-plbfz:{mountpoint:/var/lib/kubelet/pods/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e/volumes/kubernetes.io~projected/kube-api-access-plbfz major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e0f2b93-0bed-447a-99ee-5b47f3bc09bf/volumes/kubernetes.io~projected/kube-api-access-r9tww:{mountpoint:/var/lib/kubelet/pods/6e0f2b93-0bed-447a-99ee-5b47f3bc09bf/volumes/kubernetes.io~projected/kube-api-access-r9tww major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e0f2b93-0bed-447a-99ee-5b47f3bc09bf/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/6e0f2b93-0bed-447a-99ee-5b47f3bc09bf/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/778383aa-b3db-45a6-8bec-9891f7818e94/volumes/kubernetes.io~projected/kube-api-access-dt25m:{mountpoint:/var/lib/kubelet/pods/778383aa-b3db-45a6-8bec-9891f7818e94/volumes/kubernetes.io~projected/kube-api-access-dt25m major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79212f04-d10c-43ee-9dd5-41b4fd1917b2/volumes/kubernetes.io~projected/kube-api-access-djdsz:{mountpoint:/var/lib/kubelet/pods/79212f04-d10c-43ee-9dd5-41b4fd1917b2/volumes/kubernetes.io~projected/kube-api-access-djdsz major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9cf65c17-1c79-4cd5-9ef1-dceb467878b0/volumes/kubernetes.io~projected/kube-api-access-vtbgb:{mountpoint:/var/lib/kubelet/pods/9cf65c17-1c79-4cd5-9ef1-dceb467878b0/volumes/kubernetes.io~projected/kube-api-access-vtbgb major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9cf65c17-1c79-4cd5-9ef1-dceb467878b0/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/9cf65c17-1c79-4cd5-9ef1-dceb467878b0/volumes/kubernetes.io~secret/metrics-certs major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a831ae97-8f46-4b85-bfdb-59042b7b8b0c/volumes/kubernetes.io~projected/kube-api-access-mpp49:{mountpoint:/var/lib/kubelet/pods/a831ae97-8f46-4b85-bfdb-59042b7b8b0c/volumes/kubernetes.io~projected/kube-api-access-mpp49 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b57a39bc-800c-4ea8-a0ef-4a1721786b70/volumes/kubernetes.io~projected/kube-api-access-fjnd5:{mountpoint:/var/lib/kubelet/pods/b57a39bc-800c-4ea8-a0ef-4a1721786b70/volumes/kubernetes.io~projected/kube-api-access-fjnd5 major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b57a39bc-800c-4ea8-a0ef-4a1721786b70/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/b57a39bc-800c-4ea8-a0ef-4a1721786b70/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b57a39bc-800c-4ea8-a0ef-4a1721786b70/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/b57a39bc-800c-4ea8-a0ef-4a1721786b70/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b8a607f9-6133-4848-bdd6-d48e447cec86/volumes/kubernetes.io~projected/kube-api-access-2zn5r:{mountpoint:/var/lib/kubelet/pods/b8a607f9-6133-4848-bdd6-d48e447cec86/volumes/kubernetes.io~projected/kube-api-access-2zn5r major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b8a607f9-6133-4848-bdd6-d48e447cec86/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/b8a607f9-6133-4848-bdd6-d48e447cec86/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b91c00dc-c14f-4b3a-b268-1eb6dcf9ad4b/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/b91c00dc-c14f-4b3a-b268-1eb6dcf9ad4b/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b993b313-db8b-42f5-ad99-2d2ab5bb49e0/volumes/kubernetes.io~projected/kube-api-access-kq9fz:{mountpoint:/var/lib/kubelet/pods/b993b313-db8b-42f5-ad99-2d2ab5bb49e0/volumes/kubernetes.io~projected/kube-api-access-kq9fz major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c29d200a-c917-46e5-9cf5-21f4d952e470/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/c29d200a-c917-46e5-9cf5-21f4d952e470/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c29d200a-c917-46e5-9cf5-21f4d952e470/volumes/kubernetes.io~projected/kube-api-access-jssl5:{mountpoint:/var/lib/kubelet/pods/c29d200a-c917-46e5-9cf5-21f4d952e470/volumes/kubernetes.io~projected/kube-api-access-jssl5 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c29d200a-c917-46e5-9cf5-21f4d952e470/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/c29d200a-c917-46e5-9cf5-21f4d952e470/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce15fea2-6b11-4b2b-8abb-7ec29840c78f/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/ce15fea2-6b11-4b2b-8abb-7ec29840c78f/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce15fea2-6b11-4b2b-8abb-7ec29840c78f/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/ce15fea2-6b11-4b2b-8abb-7ec29840c78f/volumes/kubernetes.io~empty-dir/tmp major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce15fea2-6b11-4b2b-8abb-7ec29840c78f/volumes/kubernetes.io~projected/kube-api-access-rfmqk:{mountpoint:/var/lib/kubelet/pods/ce15fea2-6b11-4b2b-8abb-7ec29840c78f/volumes/kubernetes.io~projected/kube-api-access-rfmqk major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d505ffde-320b-41c5-a02e-374ee52e6a1a/volumes/kubernetes.io~projected/kube-api-access-njbgq:{mountpoint:/var/lib/kubelet/pods/d505ffde-320b-41c5-a02e-374ee52e6a1a/volumes/kubernetes.io~projected/kube-api-access-njbgq major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d505ffde-320b-41c5-a02e-374ee52e6a1a/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/d505ffde-320b-41c5-a02e-374ee52e6a1a/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d505ffde-320b-41c5-a02e-374ee52e6a1a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/d505ffde-320b-41c5-a02e-374ee52e6a1a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d505ffde-320b-41c5-a02e-374ee52e6a1a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/d505ffde-320b-41c5-a02e-374ee52e6a1a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d505ffde-320b-41c5-a02e-374ee52e6a1a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/d505ffde-320b-41c5-a02e-374ee52e6a1a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d505ffde-320b-41c5-a02e-374ee52e6a1a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/d505ffde-320b-41c5-a02e-374ee52e6a1a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d505ffde-320b-41c5-a02e-374ee52e6a1a/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/d505ffde-320b-41c5-a02e-374ee52e6a1a/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d59f146b-82ee-4f26-854a-cba274a89600/volumes/kubernetes.io~projected/kube-api-access-bzjx9:{mountpoint:/var/lib/kubelet/pods/d59f146b-82ee-4f26-854a-cba274a89600/volumes/kubernetes.io~projected/kube-api-access-bzjx9 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e106ef93-623a-4cb4-9848-31087f1a7711/volumes/kubernetes.io~projected/kube-api-access-cztkv:{mountpoint:/var/lib/kubelet/pods/e106ef93-623a-4cb4-9848-31087f1a7711/volumes/kubernetes.io~projected/kube-api-access-cztkv major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e106ef93-623a-4cb4-9848-31087f1a7711/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/e106ef93-623a-4cb4-9848-31087f1a7711/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:335 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/e82ab96ec3df2f54402ffaf5052c2c04278e0bb6bb821ebc1a3ca0395711006f/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/10eed12753a0f8a039e848ae603213e97c42db33ede82ff541ba64bbc6a4db27/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/505cace59a1b3abe0aa15bf25d139e8c1293feb4b4748b081ef6825f8c96ba2a/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/0e2adc21158b9e6e4a961b83e6d097b8b6d4dceb13008d21864e005429ec33ed/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/8444cda5aefde24d334b834d82a3ddddf18ca8ff6f808e00e4e9e51456cce8da/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/f5d3fc8b1c17b2b5208947b336eecee4a9fb43e258d675c6b418980478cc9f75/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/088916dfd4e5c01b63894d914598a5e182f81700e009f5dae86ce997c1c40089/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/e469d9fa222f8723c5896c471ce95a6d2d8d6d8ca23bfbbf8ba920e54fe4edff/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/7bbb66014c2910135bd477b1d2bccd9f7c210aa982338576468e2f1719aafe0d/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/a316ce644aa15d737d3d33e7963d712cdc430400f5eec8bb6e4a9aa4b7caa62d/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-145:{mountpoint:/var/lib/containers/storage/overlay/2f843e0939314b31ca6d7c2519a0b13792a0b1d302d8901e1556fc8b9e05f673/merged major:0 minor:145 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/224244a44d7fa32abf24417d3ab55ee6a6d39063a9bc8f4ae85a929794d3f159/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/30caf7ec23a5c4bec245edefc09c5fc9e38ec8e6fbb2577d1ce6ec1aa1a4255f/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/76583c3b88e5b70d4c7b65a03885be5738abd191ef110548d704f8948b937489/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/554548109a22b2b42215e2fef497e8986550081e0758a961a9e2a1ff128a08d0/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/43fc882de96ac5f6c842c7da8b6c805012146c48c169fdf7172f8caf11fcd94e/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/531ba76a60882913c552c791aa38212287d681911c88b5ad466675d5b2227001/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-174:{mountpoint:/var/lib/containers/storage/overlay/d001fa34d242b1ceef0889e9dcdbe694d366327058da7e1bc082444da1e57860/merged major:0 minor:174 fsType:overlay blockSize:0} overlay_0-179:{mountpoint:/var/lib/containers/storage/overlay/2d6eddb1cdc58710e5bac528831a6c1689620bb0cebc0b53e3095cbccabcdbae/merged major:0 minor:179 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/dae0f557573ad8e8653513ac200bbd5ec5bd4355d5c68868480f86f061de07e3/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/323f3349b7b4b969cb71810ef2f5a64fc0022a17f5755f2f67d3102fae1dfed2/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/a98099cbb40bddc4d63cfe30bacf1e54f868c6c1ad5b730d07217bce9c83df66/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/34966f10f5d6e21a888552c403c336c08d1a4af2ca442a7b01e60e34caa2bb2a/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/fff1e1e7d5fc684648704d9793bcc350723f6c65c70c07f23ee9a51cc53fd6db/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/14c3394a724fd401a38c7b5f655f57b01d6f7b88f5102046cc49096b32731f2f/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/c506f0c40bde6dce24e982651972836936721c3f4cdc5904910c7e900d0ca6c5/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/e13dc1bc4a450203600d19b8dff2f23407216de9665d40c4fdea9a8eb088a9c3/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/2027e76ca1aa6c2602bf1fd7f0ef74e0094be24a21474c82ea96d40d3b8d52e3/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/c9a81d55e87f3369c37fa4dd7d19e0400a87f388734d773e01cf42d7cf20c4f9/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-239:{mountpoint:/var/lib/containers/storage/overlay/c2fba56457e78d92ffa69af999f654e04be2fa26ed9ebad6de91227f72aee54e/merged major:0 minor:239 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/97aca8c490dbb9d19549dbe5cb4f6ed8bce354ef53d1d3f8a72aa9e8cdd69400/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/b78ea8acb0a22dc1f16c700ae963872419d8dd6989413e4915fba1a59e46ea28/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/70743c024b44897fc0b6df0351ffa109ff633ce2ab0b696eff43caedba35c16f/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/7e5c1f95b7218d2727792d21b7191c0df7dd98e94c913344bd32aef8c662610d/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/b070d81c30e6d4c485b5bded51a5e801e5cd509e9fc30191b4d277535c41a535/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/f8d20b45b22a3fc2bc71c403fdab290dbc457e77e13197705974adae9b19e9fe/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/3d1a9aa869795870aff23982197e12ccfcd019df486a8b06a7a59012aaea40a9/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/5791070f8696d71dca3d4d24a44e838a66bd2143af6c2d8bc732ac5f96b3743d/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/85ca36ff998f38b816b99efdfe95dd035cc77afb8325c8c42b2a5a279425b82d/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/8d0eb4e4287d98849424494bcca75daf3e0a1c538f2afdc6860f41b35e5cb287/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/383e245796a112602c2b1802a329aaeec3e84d95858a4887f1039a3d76f267a1/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/dbd0a308d37849edccfa766dbfab123cfab3b93b119947638e21b903588333b3/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/cdebae18b6ed7293ef7509e2bd63fb0810189ecc7ff0df69ae186618fc713a5f/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-325:{mountpoint:/var/lib/containers/storage/overlay/ef9efe6b1b65821aac1aa9502ac679b669223d94982c40c560f08e44e1edf94f/merged major:0 minor:325 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/5ca7c97fa474ef5c6f04f0427bc2df69960afbecc9fde4e615acf4782491ea29/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/bfc106f0f97a3637e3294b33a0e58cdf7a8a714722f2fb1e4a3699d5a49b574c/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/4ce6996461a6b64495b46b4c167e23fcf9ef86d302483a066759f95b2f5ad53b/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/f2547c9d4816e579ec0b572eae8534af73533cafa7466fe712e583987b7821c3/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/3f51e5b94909d9a14995a1c96b87de67934be6f422585cc590f72e4768b605d3/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/389fe97500e2499950ead328eb73ec04610f7d3b5260467a8d4bf16d99f3fcaf/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/7ec5b3c21326105129d468f2abf08888b07cb4f8ee58e4f093e145b509650bb1/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-367:{mountpoint:/var/lib/containers/storage/overlay/bf109c9aa9b6024926dcc47b75759f110b38ba9ed4481bfccf09b3a04368ec5d/merged major:0 minor:367 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/e64485411bb00c9fb80f89976d39094d6cbba9b53af9e77b70bea81adb68cc29/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/1d6a273ac507e1498d1407f1f154f48aff3f345c7dda10cff3dcb0ab8ec99497/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/2ffd3f12c8bb220ccacd5878b0b140ce7f00372886d2c16968ffa37e2f5b160c/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-75:{mountpoint:/var/lib/containers/storage/overlay/ebcb8e9120f3146eac8c4974ecbd5ba29abaf7ea8ce1b314974cd5fe29a09eb0/merged major:0 minor:75 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/196bbc5db8078e8fddf713decdd96a70c55b711f209deee02d1dba736a38791c/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/f6f493dbd4274a25e40965b179f48a668cba383910ec2ac8198fb04f6132d01a/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/e9b9f7ea5ee20fa32b2c77a04baf14e185aaecaded54499ba939b4c9fec9ae7f/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/f781dcee6345ce365dcb7acd6ba13bd298a017b28b9ff965710b6025eccb6a77/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/f2345430963c876d32d7a081a14e421ece8d9906d5f7f0b95816d7d03faa815b/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/24a1c4f625898507715d8bab5b848c8b5049eed99c958035d53475027ca27dcf/merged major:0 minor:98 fsType:overlay blockSize:0}]
Nov 03 17:46:01.047281 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.044350    5315 manager.go:217] Machine: {Timestamp:2026-11-03 17:46:01.043486046 +0000 UTC m=+0.059684111 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099633 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25fb36af538e17c812be0d5d125f23 SystemUUID:ec25fb36-af53-8e17-c812-be0d5d125f23 BootID:1fe403c3-07d4-45e7-8194-0505140c7d13 Filesystems:[{Device:/var/lib/kubelet/pods/ce15fea2-6b11-4b2b-8abb-7ec29840c78f/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:50 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/92784a8e82349f46b51d0d81cfa3ba97b2d371b63d606eaf1c457b088f74c2cd/userdata/shm DeviceMajor:0 DeviceMinor:346 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-325 DeviceMajor:0 DeviceMinor:325 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-179 DeviceMajor:0 DeviceMinor:179 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d505ffde-320b-41c5-a02e-374ee52e6a1a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:286 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d505ffde-320b-41c5-a02e-374ee52e6a1a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:288 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/d505ffde-320b-41c5-a02e-374ee52e6a1a/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:287 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b57a39bc-800c-4ea8-a0ef-4a1721786b70/volumes/kubernetes.io~projected/kube-api-access-fjnd5 DeviceMajor:0 DeviceMinor:230 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a493b902a49a44f6003be191f76a551ca141c08c0d1ece763fbf1d170f15e03b/userdata/shm DeviceMajor:0 DeviceMinor:292 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/79212f04-d10c-43ee-9dd5-41b4fd1917b2/volumes/kubernetes.io~projected/kube-api-access-djdsz DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3e881e23-d118-4245-b855-b1322d133ef1/volumes/kubernetes.io~projected/kube-api-access-9jsj9 DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9cf65c17-1c79-4cd5-9ef1-dceb467878b0/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:340 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-174 DeviceMajor:0 DeviceMinor:174 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/55b54081-644c-464e-91e1-15c2d8f5be59/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:265 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c29d200a-c917-46e5-9cf5-21f4d952e470/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:49 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b8a607f9-6133-4848-bdd6-d48e447cec86/volumes/kubernetes.io~projected/kube-api-access-2zn5r DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/139653ececacf7123894d53780a05a97b3a3c808ff74c54de72556fa79a00dfa/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ce15fea2-6b11-4b2b-8abb-7ec29840c78f/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:48 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/abb3d20d17c9e1057141552b05e0c65826502f65fcb07c44d2c74c00e481b7f1/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/55b54081-644c-464e-91e1-15c2d8f5be59/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:269 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d505ffde-320b-41c5-a02e-374ee52e6a1a/volumes/kubernetes.io~projected/kube-api-access-njbgq DeviceMajor:0 DeviceMinor:291 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/68856dcdd260964150b8ee215612e33d4f82033dbfa90a19ac4dbea63d6c3330/userdata/shm DeviceMajor:0 DeviceMinor:298 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d59f146b-82ee-4f26-854a-cba274a89600/volumes/kubernetes.io~projected/kube-api-access-bzjx9 DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-367 DeviceMajor:0 DeviceMinor:367 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b91c00dc-c14f-4b3a-b268-1eb6dcf9ad4b/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1794a07f998f56a2861ae8bc703815fb34354f4e54a42e590f6a7535ced033d9/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/00bbb7d679385391ad209f99b21a55fceb70b6ed883a692a74de60e11e9024d6/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3ad8db60b8760552cac5c6731018d7b4d80e8e8faf4944fcd6e0ec4fc4fdad1a/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d505ffde-320b-41c5-a02e-374ee52e6a1a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:282 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e/volumes/kubernetes.io~projected/kube-api-access-plbfz DeviceMajor:0 DeviceMinor:54 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b993b313-db8b-42f5-ad99-2d2ab5bb49e0/volumes/kubernetes.io~projected/kube-api-access-kq9fz DeviceMajor:0 DeviceMinor:53 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-239 DeviceMajor:0 DeviceMinor:239 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/20eb0bd6f5eabf540587d2a2680526d0bf218614cb1ae6e6466f48b069f0dfad/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b8a607f9-6133-4848-bdd6-d48e447cec86/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0bb1fb1187d8ea80dd6c6dda38067a5c154fb2c249c4c69851fcc12aab410f85/userdata/shm DeviceMajor:0 DeviceMinor:204 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/735b63977e239842f058114ff665b7db5d04b38dd6ab509212b94544eaee250d/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-75 DeviceMajor:0 DeviceMinor:75 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e6269ad31b395b408406d12e3bfbab8b2317923cdf821b7c815c69caac58878a/userdata/shm DeviceMajor:0 DeviceMinor:73 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6e0f2b93-0bed-447a-99ee-5b47f3bc09bf/volumes/kubernetes.io~projected/kube-api-access-r9tww DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/778383aa-b3db-45a6-8bec-9891f7818e94/volumes/kubernetes.io~projected/kube-api-access-dt25m DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8549638c99969e115d70c745d0c81574071f2c1ac772f71c1dff07aaa52634aa/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b57a39bc-800c-4ea8-a0ef-4a1721786b70/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:229 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2d40add9b19ed573199dbc143ab57a05279318c4f443a5f8ccd193dbed28b065/userdata/shm DeviceMajor:0 DeviceMinor:344 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/919fb8112573214d30fe24a1e5472e0723bcc1c57c5a2f3b98c69c3de9573523/userdata/shm DeviceMajor:0 DeviceMinor:271 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e106ef93-623a-4cb4-9848-31087f1a7711/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:297 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/548bb6c5c09ad0038c341aa2088a3cd11b1911863d21f19dc91b1d728a093136/userdata/shm DeviceMajor:0 DeviceMinor:342 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/55b54081-644c-464e-91e1-15c2d8f5be59/volumes/kubernetes.io~projected/kube-api-access-2jstr DeviceMajor:0 DeviceMinor:270 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9cf65c17-1c79-4cd5-9ef1-dceb467878b0/volumes/kubernetes.io~projected/kube-api-access-vtbgb DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ce15fea2-6b11-4b2b-8abb-7ec29840c78f/volumes/kubernetes.io~projected/kube-api-access-rfmqk DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3e881e23-d118-4245-b855-b1322d133ef1/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e106ef93-623a-4cb4-9848-31087f1a7711/volumes/kubernetes.io~projected/kube-api-access-cztkv DeviceMajor:0 DeviceMinor:296 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/43c9da28-334d-4bb3-a6bc-e3da2d561633/volumes/kubernetes.io~projected/kube-api-access-v89s6 DeviceMajor:0 DeviceMinor:341 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-145 DeviceMajor:0 DeviceMinor:145 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a831ae97-8f46-4b85-bfdb-59042b7b8b0c/volumes/kubernetes.io~projected/kube-api-access-mpp49 DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7bd829a4c167033d557935b56396132797cbcee937ae350c15682fc836a324b2/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:335 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6e0f2b93-0bed-447a-99ee-5b47f3bc09bf/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d505ffde-320b-41c5-a02e-374ee52e6a1a/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:289 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d505ffde-320b-41c5-a02e-374ee52e6a1a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:290 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/17f5aed41c2adc3c7122f2af59303d8dd656fbdcfaba6b7b85cdd74ede8e5804/userdata/shm DeviceMajor:0 DeviceMinor:52 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e30ab61f1fcccd658bc913224880d4de5f884763621080cf78cb3880b3cef69f/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b57a39bc-800c-4ea8-a0ef-4a1721786b70/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c29d200a-c917-46e5-9cf5-21f4d952e470/volumes/kubernetes.io~projected/kube-api-access-jssl5 DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e9ad1fb5061bef579da494a3748707e9bc6214af69ee74945e6254c3ce22365f/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c29d200a-c917-46e5-9cf5-21f4d952e470/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0bb1fb1187d8ea8 MacAddress:b2:0d:27:52:ac:87 Speed:10000 Mtu:8901} {Name:139653ececacf71 MacAddress:ae:3d:78:81:d6:a7 Speed:10000 Mtu:8901} {Name:20eb0bd6f5eabf5 MacAddress:5a:2a:54:02:a2:1a Speed:10000 Mtu:8901} {Name:2d40add9b19ed57 MacAddress:2a:67:5a:36:d5:56 Speed:10000 Mtu:8901} {Name:548bb6c5c09ad00 MacAddress:de:90:12:57:f7:a4 Speed:10000 Mtu:8901} {Name:68856dcdd260964 MacAddress:8e:c8:2a:40:d1:d5 Speed:10000 Mtu:8901} {Name:7bd829a4c167033 MacAddress:1a:90:0b:7c:ba:13 Speed:10000 Mtu:8901} {Name:92784a8e82349f4 MacAddress:b2:51:66:6b:2c:d5 Speed:10000 Mtu:8901} {Name:a493b902a49a44f MacAddress:12:b7:5a:4c:31:45 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:c5:e1:7a:8b Speed:0 Mtu:9001} {Name:br-int MacAddress:d6:c2:86:88:db:1d Speed:0 Mtu:8901} {Name:ens5 MacAddress:06:ff:c5:e1:7a:8b Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:b6:ee:5d:82:d5:30 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:aa:e4:c3:dd:f3:b0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 17:46:01.047281 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.044866    5315 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 17:46:01.047281 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.044932    5315 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 17:46:01.047281 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.045228    5315 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 17:46:01.047281 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.045249    5315 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-204.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 17:46:01.047281 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.045369    5315 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 17:46:01.047281 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.045377    5315 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 17:46:01.047281 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.045391    5315 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 17:46:01.047281 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.045410    5315 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 17:46:01.047281 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.045488    5315 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 17:46:01.047281 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.045654    5315 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 17:46:01.047281 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.045765    5315 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 17:46:01.047281 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.045799    5315 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 17:46:01.047281 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.045811    5315 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 17:46:01.047281 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.045820    5315 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 17:46:01.047281 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.045829    5315 kubelet.go:405] "Adding apiserver pod source"
Nov 03 17:46:01.047281 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.045838    5315 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 17:46:01.047281 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.046537    5315 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 17:46:01.047281 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.046598    5315 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 17:46:01.047281 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.047078    5315 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 17:46:01.049327 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.049306    5315 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 17:46:01.049327 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.049321    5315 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 17:46:01.049487 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.049418    5315 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 03 17:46:01.049622 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.049595    5315 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 17:46:01.049667 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.049646    5315 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 17:46:01.049747 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.049738    5315 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 17:46:01.049787 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.049752    5315 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 17:46:01.049787 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.049759    5315 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 17:46:01.049787 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.049765    5315 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 17:46:01.049787 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.049770    5315 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 17:46:01.049787 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.049776    5315 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 17:46:01.049787 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.049781    5315 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 17:46:01.049787 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.049787    5315 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 17:46:01.050125 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.049794    5315 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 17:46:01.050125 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.049800    5315 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 17:46:01.050125 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.049814    5315 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 17:46:01.050125 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.049823    5315 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 17:46:01.050125 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.049848    5315 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 17:46:01.050125 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.049853    5315 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 17:46:01.053176 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.053161    5315 server.go:1263] "Started kubelet"
Nov 03 17:46:01.053624 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.053553    5315 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 17:46:01.054434 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.053721    5315 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 17:46:01.054434 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.054096    5315 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 17:46:01.053767 ip-10-0-128-204 systemd[1]: Started Kubernetes Kubelet.
Nov 03 17:46:01.054668 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.054524    5315 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 17:46:01.054866 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.054849    5315 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 17:46:01.059006 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.058986    5315 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 17:46:01.060255 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.059599    5315 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 17:46:01.060344 ip-10-0-128-204 kubenswrapper[5315]: E1103 17:46:01.060324    5315 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-204.us-west-2.compute.internal\" not found"
Nov 03 17:46:01.060344 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.060327    5315 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 17:39:50 +0000 UTC" deadline="2028-07-04 20:38:58.774525759 +0000 UTC"
Nov 03 17:46:01.060451 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.060353    5315 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14618h52m57.714176101s"
Nov 03 17:46:01.060451 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.060370    5315 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 17:46:01.060451 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.060378    5315 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 17:46:01.060581 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.060470    5315 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 17:46:01.062309 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.061035    5315 factory.go:55] Registering systemd factory
Nov 03 17:46:01.062309 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.061060    5315 factory.go:223] Registration of the systemd container factory successfully
Nov 03 17:46:01.062309 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.062006    5315 factory.go:153] Registering CRI-O factory
Nov 03 17:46:01.062309 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.062022    5315 factory.go:223] Registration of the crio container factory successfully
Nov 03 17:46:01.062309 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.062084    5315 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 17:46:01.062309 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.062103    5315 factory.go:103] Registering Raw factory
Nov 03 17:46:01.062309 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.062125    5315 manager.go:1199] Started watching for new ooms in manager
Nov 03 17:46:01.062706 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.062493    5315 manager.go:319] Starting recovery of all containers
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.063972    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b57a39bc-800c-4ea8-a0ef-4a1721786b70" volumeName="kubernetes.io/secret/b57a39bc-800c-4ea8-a0ef-4a1721786b70-prometheus-operator-tls" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064039    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b57a39bc-800c-4ea8-a0ef-4a1721786b70" volumeName="kubernetes.io/secret/b57a39bc-800c-4ea8-a0ef-4a1721786b70-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064057    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d59f146b-82ee-4f26-854a-cba274a89600" volumeName="kubernetes.io/projected/d59f146b-82ee-4f26-854a-cba274a89600-kube-api-access-bzjx9" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064079    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e881e23-d118-4245-b855-b1322d133ef1" volumeName="kubernetes.io/configmap/3e881e23-d118-4245-b855-b1322d133ef1-config-volume" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064094    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e881e23-d118-4245-b855-b1322d133ef1" volumeName="kubernetes.io/projected/3e881e23-d118-4245-b855-b1322d133ef1-kube-api-access-9jsj9" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064108    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55b54081-644c-464e-91e1-15c2d8f5be59" volumeName="kubernetes.io/configmap/55b54081-644c-464e-91e1-15c2d8f5be59-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064120    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e0f2b93-0bed-447a-99ee-5b47f3bc09bf" volumeName="kubernetes.io/secret/6e0f2b93-0bed-447a-99ee-5b47f3bc09bf-cert" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064132    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79212f04-d10c-43ee-9dd5-41b4fd1917b2" volumeName="kubernetes.io/configmap/79212f04-d10c-43ee-9dd5-41b4fd1917b2-serviceca" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064147    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b91c00dc-c14f-4b3a-b268-1eb6dcf9ad4b" volumeName="kubernetes.io/secret/b91c00dc-c14f-4b3a-b268-1eb6dcf9ad4b-agent-certs" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064163    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b993b313-db8b-42f5-ad99-2d2ab5bb49e0" volumeName="kubernetes.io/configmap/b993b313-db8b-42f5-ad99-2d2ab5bb49e0-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064176    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce15fea2-6b11-4b2b-8abb-7ec29840c78f" volumeName="kubernetes.io/projected/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-kube-api-access-rfmqk" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064193    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a831ae97-8f46-4b85-bfdb-59042b7b8b0c" volumeName="kubernetes.io/empty-dir/a831ae97-8f46-4b85-bfdb-59042b7b8b0c-tmp-dir" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064205    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b993b313-db8b-42f5-ad99-2d2ab5bb49e0" volumeName="kubernetes.io/configmap/b993b313-db8b-42f5-ad99-2d2ab5bb49e0-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064222    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce15fea2-6b11-4b2b-8abb-7ec29840c78f" volumeName="kubernetes.io/empty-dir/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-tmp" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064240    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e" volumeName="kubernetes.io/configmap/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-cni-binary-copy" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064255    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="778383aa-b3db-45a6-8bec-9891f7818e94" volumeName="kubernetes.io/configmap/778383aa-b3db-45a6-8bec-9891f7818e94-iptables-alerter-script" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064268    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d505ffde-320b-41c5-a02e-374ee52e6a1a" volumeName="kubernetes.io/secret/d505ffde-320b-41c5-a02e-374ee52e6a1a-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064280    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d505ffde-320b-41c5-a02e-374ee52e6a1a" volumeName="kubernetes.io/secret/d505ffde-320b-41c5-a02e-374ee52e6a1a-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064309    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8a607f9-6133-4848-bdd6-d48e447cec86" volumeName="kubernetes.io/empty-dir/b8a607f9-6133-4848-bdd6-d48e447cec86-data-volume" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064323    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c29d200a-c917-46e5-9cf5-21f4d952e470" volumeName="kubernetes.io/projected/c29d200a-c917-46e5-9cf5-21f4d952e470-kube-api-access-jssl5" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064335    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d505ffde-320b-41c5-a02e-374ee52e6a1a" volumeName="kubernetes.io/configmap/d505ffde-320b-41c5-a02e-374ee52e6a1a-metrics-client-ca" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064349    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43c9da28-334d-4bb3-a6bc-e3da2d561633" volumeName="kubernetes.io/projected/43c9da28-334d-4bb3-a6bc-e3da2d561633-kube-api-access-v89s6" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064363    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55b54081-644c-464e-91e1-15c2d8f5be59" volumeName="kubernetes.io/secret/55b54081-644c-464e-91e1-15c2d8f5be59-node-exporter-tls" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064385    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55b54081-644c-464e-91e1-15c2d8f5be59" volumeName="kubernetes.io/secret/55b54081-644c-464e-91e1-15c2d8f5be59-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064397    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8a607f9-6133-4848-bdd6-d48e447cec86" volumeName="kubernetes.io/secret/b8a607f9-6133-4848-bdd6-d48e447cec86-insights-runtime-extractor-tls" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064410    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c29d200a-c917-46e5-9cf5-21f4d952e470" volumeName="kubernetes.io/configmap/c29d200a-c917-46e5-9cf5-21f4d952e470-ovnkube-script-lib" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064424    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d505ffde-320b-41c5-a02e-374ee52e6a1a" volumeName="kubernetes.io/empty-dir/d505ffde-320b-41c5-a02e-374ee52e6a1a-tmp" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064439    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="778383aa-b3db-45a6-8bec-9891f7818e94" volumeName="kubernetes.io/projected/778383aa-b3db-45a6-8bec-9891f7818e94-kube-api-access-dt25m" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064457    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e106ef93-623a-4cb4-9848-31087f1a7711" volumeName="kubernetes.io/projected/e106ef93-623a-4cb4-9848-31087f1a7711-kube-api-access-cztkv" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064472    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e881e23-d118-4245-b855-b1322d133ef1" volumeName="kubernetes.io/empty-dir/3e881e23-d118-4245-b855-b1322d133ef1-tmp-dir" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064497    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a831ae97-8f46-4b85-bfdb-59042b7b8b0c" volumeName="kubernetes.io/projected/a831ae97-8f46-4b85-bfdb-59042b7b8b0c-kube-api-access-mpp49" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064520    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b57a39bc-800c-4ea8-a0ef-4a1721786b70" volumeName="kubernetes.io/projected/b57a39bc-800c-4ea8-a0ef-4a1721786b70-kube-api-access-fjnd5" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064532    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8a607f9-6133-4848-bdd6-d48e447cec86" volumeName="kubernetes.io/configmap/b8a607f9-6133-4848-bdd6-d48e447cec86-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064545    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e881e23-d118-4245-b855-b1322d133ef1" volumeName="kubernetes.io/secret/3e881e23-d118-4245-b855-b1322d133ef1-metrics-tls" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064556    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55b54081-644c-464e-91e1-15c2d8f5be59" volumeName="kubernetes.io/configmap/55b54081-644c-464e-91e1-15c2d8f5be59-metrics-client-ca" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064567    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79212f04-d10c-43ee-9dd5-41b4fd1917b2" volumeName="kubernetes.io/projected/79212f04-d10c-43ee-9dd5-41b4fd1917b2-kube-api-access-djdsz" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064588    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c29d200a-c917-46e5-9cf5-21f4d952e470" volumeName="kubernetes.io/configmap/c29d200a-c917-46e5-9cf5-21f4d952e470-env-overrides" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064600    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce15fea2-6b11-4b2b-8abb-7ec29840c78f" volumeName="kubernetes.io/empty-dir/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-etc-tuned" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064630    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d505ffde-320b-41c5-a02e-374ee52e6a1a" volumeName="kubernetes.io/projected/d505ffde-320b-41c5-a02e-374ee52e6a1a-kube-api-access-njbgq" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064642    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d505ffde-320b-41c5-a02e-374ee52e6a1a" volumeName="kubernetes.io/secret/d505ffde-320b-41c5-a02e-374ee52e6a1a-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064668    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d505ffde-320b-41c5-a02e-374ee52e6a1a" volumeName="kubernetes.io/secret/d505ffde-320b-41c5-a02e-374ee52e6a1a-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064680    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cf65c17-1c79-4cd5-9ef1-dceb467878b0" volumeName="kubernetes.io/projected/9cf65c17-1c79-4cd5-9ef1-dceb467878b0-kube-api-access-vtbgb" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064692    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55b54081-644c-464e-91e1-15c2d8f5be59" volumeName="kubernetes.io/projected/55b54081-644c-464e-91e1-15c2d8f5be59-kube-api-access-2jstr" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064704    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cf65c17-1c79-4cd5-9ef1-dceb467878b0" volumeName="kubernetes.io/secret/9cf65c17-1c79-4cd5-9ef1-dceb467878b0-metrics-certs" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064715    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e0f2b93-0bed-447a-99ee-5b47f3bc09bf" volumeName="kubernetes.io/projected/6e0f2b93-0bed-447a-99ee-5b47f3bc09bf-kube-api-access-r9tww" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064728    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b91c00dc-c14f-4b3a-b268-1eb6dcf9ad4b" volumeName="kubernetes.io/configmap/b91c00dc-c14f-4b3a-b268-1eb6dcf9ad4b-konnectivity-ca" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064740    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b993b313-db8b-42f5-ad99-2d2ab5bb49e0" volumeName="kubernetes.io/configmap/b993b313-db8b-42f5-ad99-2d2ab5bb49e0-cni-binary-copy" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064750    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b993b313-db8b-42f5-ad99-2d2ab5bb49e0" volumeName="kubernetes.io/projected/b993b313-db8b-42f5-ad99-2d2ab5bb49e0-kube-api-access-kq9fz" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064761    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c29d200a-c917-46e5-9cf5-21f4d952e470" volumeName="kubernetes.io/configmap/c29d200a-c917-46e5-9cf5-21f4d952e470-ovnkube-config" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064773    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e" volumeName="kubernetes.io/configmap/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-multus-daemon-config" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064785    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e" volumeName="kubernetes.io/projected/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-kube-api-access-plbfz" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064797    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b57a39bc-800c-4ea8-a0ef-4a1721786b70" volumeName="kubernetes.io/configmap/b57a39bc-800c-4ea8-a0ef-4a1721786b70-metrics-client-ca" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064809    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c29d200a-c917-46e5-9cf5-21f4d952e470" volumeName="kubernetes.io/secret/c29d200a-c917-46e5-9cf5-21f4d952e470-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064821    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d505ffde-320b-41c5-a02e-374ee52e6a1a" volumeName="kubernetes.io/secret/d505ffde-320b-41c5-a02e-374ee52e6a1a-secret-thanos-querier-tls" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064833    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30" volumeName="kubernetes.io/secret/f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30-original-pull-secret" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064844    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55b54081-644c-464e-91e1-15c2d8f5be59" volumeName="kubernetes.io/empty-dir/55b54081-644c-464e-91e1-15c2d8f5be59-node-exporter-textfile" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064865    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e106ef93-623a-4cb4-9848-31087f1a7711" volumeName="kubernetes.io/secret/e106ef93-623a-4cb4-9848-31087f1a7711-monitoring-plugin-cert" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064879    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8a607f9-6133-4848-bdd6-d48e447cec86" volumeName="kubernetes.io/projected/b8a607f9-6133-4848-bdd6-d48e447cec86-kube-api-access-2zn5r" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064891    5315 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d505ffde-320b-41c5-a02e-374ee52e6a1a" volumeName="kubernetes.io/secret/d505ffde-320b-41c5-a02e-374ee52e6a1a-secret-grpc-tls" seLinuxMountContext=""
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064903    5315 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 17:46:01.067544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.064911    5315 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 17:46:01.085896 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.085863    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-97p65_c29d200a-c917-46e5-9cf5-21f4d952e470/ovn-acl-logging/0.log"
Nov 03 17:46:01.086336 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.086313    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-97p65_c29d200a-c917-46e5-9cf5-21f4d952e470/ovn-acl-logging/0.log"
Nov 03 17:46:01.095015 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.094981    5315 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 17:46:01.096269 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.096248    5315 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 17:46:01.096269 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.096273    5315 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 17:46:01.096438 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.096296    5315 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 17:46:01.096438 ip-10-0-128-204 kubenswrapper[5315]: E1103 17:46:01.096337    5315 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 17:46:01.104494 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.104473    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-97p65_c29d200a-c917-46e5-9cf5-21f4d952e470/ovn-acl-logging/0.log"
Nov 03 17:46:01.107072 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.107040    5315 generic.go:356] "Generic (PLEG): container finished" podID="c29d200a-c917-46e5-9cf5-21f4d952e470" containerID="a068e219a50a01990dd55ec22bf5723658419d68bca038fbea25f88e779d0e78" exitCode=1
Nov 03 17:46:01.114239 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.114207    5315 generic.go:356] "Generic (PLEG): container finished" podID="55b54081-644c-464e-91e1-15c2d8f5be59" containerID="46b0f51171181de71c40b397b90fde72284af2e29fb6d4d4729af48dca5be168" exitCode=0
Nov 03 17:46:01.129866 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.129835    5315 generic.go:356] "Generic (PLEG): container finished" podID="8a90b512115a10076b69387005e99589" containerID="610f607107d75973ef19289a402337ab681768975df03a11d39328fb87704041" exitCode=0
Nov 03 17:46:01.142594 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.142567    5315 generic.go:356] "Generic (PLEG): container finished" podID="b993b313-db8b-42f5-ad99-2d2ab5bb49e0" containerID="d2f95a9ec0a2b48e2b53dfa49edb2caff146bbf1357f5793bf45ca6be0372834" exitCode=0
Nov 03 17:46:01.142594 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.142592    5315 generic.go:356] "Generic (PLEG): container finished" podID="b993b313-db8b-42f5-ad99-2d2ab5bb49e0" containerID="641f560f56ab02e85090e4a58030382c328364cd9caab180caad9fee6767d835" exitCode=0
Nov 03 17:46:01.142594 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.142598    5315 generic.go:356] "Generic (PLEG): container finished" podID="b993b313-db8b-42f5-ad99-2d2ab5bb49e0" containerID="664ccb3c5f6b0ffebf87cfce1d29ebe7d6f202ab00bc14b157430c00f377fd75" exitCode=0
Nov 03 17:46:01.142859 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.142603    5315 generic.go:356] "Generic (PLEG): container finished" podID="b993b313-db8b-42f5-ad99-2d2ab5bb49e0" containerID="84d85ef08a77cf63d35185a436052f4646f0d83e500ece74262104ed17661fba" exitCode=0
Nov 03 17:46:01.142859 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.142628    5315 generic.go:356] "Generic (PLEG): container finished" podID="b993b313-db8b-42f5-ad99-2d2ab5bb49e0" containerID="1014f827db955aeaea5b0c6ff4029b5aa0c33a5e0d581a5770fb435e2686be13" exitCode=0
Nov 03 17:46:01.142859 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.142634    5315 generic.go:356] "Generic (PLEG): container finished" podID="b993b313-db8b-42f5-ad99-2d2ab5bb49e0" containerID="8c33a28ab106380469ba6734489f9bb1b0116dc8b9a23d5a1fab50d38f6a2ee2" exitCode=0
Nov 03 17:46:01.156019 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.155990    5315 manager.go:324] Recovery completed
Nov 03 17:46:01.160436 ip-10-0-128-204 kubenswrapper[5315]: E1103 17:46:01.160379    5315 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-204.us-west-2.compute.internal\" not found"
Nov 03 17:46:01.173998 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.173980    5315 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:46:01.177707 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.177684    5315 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-204.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:46:01.177774 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.177716    5315 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-204.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:46:01.177774 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.177731    5315 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-204.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:46:01.180550 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.180534    5315 cpu_manager.go:226] "Starting" policy="none"
Nov 03 17:46:01.180550 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.180546    5315 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 17:46:01.180670 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.180550    5315 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:46:01.180670 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.180563    5315 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 17:46:01.180736 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.180708    5315 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 03 17:46:01.180736 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.180721    5315 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 03 17:46:01.180852 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.180842    5315 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 03 17:46:01.180882 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.180853    5315 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 03 17:46:01.180882 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.180859    5315 policy_none.go:50] "Start"
Nov 03 17:46:01.180882 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.180868    5315 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 17:46:01.180882 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.180878    5315 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 17:46:01.181540 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.181519    5315 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-204.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:46:01.181600 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.181552    5315 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-204.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:46:01.181600 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.181566    5315 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-204.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:46:01.181678 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.181638    5315 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 03 17:46:01.181678 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.181656    5315 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 03 17:46:01.181678 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.181665    5315 policy_none.go:44] "Start"
Nov 03 17:46:01.185795 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.185771    5315 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 17:46:01.188295 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.188283    5315 manager.go:341] "Starting Device Plugin manager"
Nov 03 17:46:01.188336 ip-10-0-128-204 kubenswrapper[5315]: E1103 17:46:01.188322    5315 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 17:46:01.188336 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.188331    5315 server.go:87] "Starting device plugin registration server"
Nov 03 17:46:01.188585 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.188576    5315 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 17:46:01.188646 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.188589    5315 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 17:46:01.188696 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.188673    5315 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 17:46:01.188811 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.188771    5315 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 17:46:01.188915 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.188818    5315 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 17:46:01.188915 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.188825    5315 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 17:46:01.189014 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.188860    5315 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T17:46:01.18880739Z","UUID":"f370f625-5c3e-4535-a5e6-5b4ebc877d7a","Handler":null,"Name":"","Endpoint":""}
Nov 03 17:46:01.190542 ip-10-0-128-204 kubenswrapper[5315]: E1103 17:46:01.190517    5315 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 17:46:01.190684 ip-10-0-128-204 kubenswrapper[5315]: E1103 17:46:01.190575    5315 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-204.us-west-2.compute.internal\" not found"
Nov 03 17:46:01.190684 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.190637    5315 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 17:46:01.190684 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.190677    5315 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 17:46:01.196984 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.196418    5315 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-204.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-204.us-west-2.compute.internal"]
Nov 03 17:46:01.197084 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.197071    5315 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:46:01.198041 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.198024    5315 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-204.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:46:01.198143 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.198049    5315 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-204.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:46:01.198143 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.198061    5315 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-204.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:46:01.198354 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.198340    5315 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:46:01.198496 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.198482    5315 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:46:01.199353 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.199337    5315 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-204.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:46:01.199444 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.199368    5315 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-204.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:46:01.199444 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.199384    5315 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-204.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:46:01.199444 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.199335    5315 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-204.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:46:01.199444 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.199414    5315 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-204.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:46:01.199444 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.199428    5315 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-204.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:46:01.199701 ip-10-0-128-204 kubenswrapper[5315]: E1103 17:46:01.199574    5315 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-204.us-west-2.compute.internal\" not found" node="ip-10-0-128-204.us-west-2.compute.internal"
Nov 03 17:46:01.199750 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.199722    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-204.us-west-2.compute.internal" event={"ID":"8dbd524262c9b524921d913509ef36fa","Type":"ContainerStarted","Data":"b07f87172a975d93912a71a3d2ab236f4ad0063091498841cad16ef0f24c6b2b"}
Nov 03 17:46:01.199790 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.199760    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-204.us-west-2.compute.internal" event={"ID":"8dbd524262c9b524921d913509ef36fa","Type":"ContainerStarted","Data":"735b63977e239842f058114ff665b7db5d04b38dd6ab509212b94544eaee250d"}
Nov 03 17:46:01.199842 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.199829    5315 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:46:01.199894 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.199825    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-204.us-west-2.compute.internal" event={"ID":"8a90b512115a10076b69387005e99589","Type":"ContainerStarted","Data":"2636ed70d99c558451ee0631f4094289029280c136380ade1436ead2b49a1d49"}
Nov 03 17:46:01.199894 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.199880    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-204.us-west-2.compute.internal" event={"ID":"8a90b512115a10076b69387005e99589","Type":"ContainerDied","Data":"610f607107d75973ef19289a402337ab681768975df03a11d39328fb87704041"}
Nov 03 17:46:01.199977 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.199901    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-204.us-west-2.compute.internal" event={"ID":"8a90b512115a10076b69387005e99589","Type":"ContainerStarted","Data":"abb3d20d17c9e1057141552b05e0c65826502f65fcb07c44d2c74c00e481b7f1"}
Nov 03 17:46:01.200822 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.200804    5315 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-204.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:46:01.200905 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.200837    5315 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-204.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:46:01.200905 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.200853    5315 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-204.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:46:01.200999 ip-10-0-128-204 kubenswrapper[5315]: E1103 17:46:01.200986    5315 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-204.us-west-2.compute.internal\" not found" node="ip-10-0-128-204.us-west-2.compute.internal"
Nov 03 17:46:01.244461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.244433    5315 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:46:01.245124 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.245103    5315 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:46:01.245234 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.245189    5315 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 03 17:46:01.245289 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.245272    5315 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:46:01.261328 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.261309    5315 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-204.us-west-2.compute.internal"
Nov 03 17:46:01.290383 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.290358    5315 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:46:01.291572 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.291552    5315 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-204.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:46:01.291671 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.291591    5315 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-204.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:46:01.291671 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.291622    5315 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-204.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:46:01.291671 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.291671    5315 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-204.us-west-2.compute.internal"
Nov 03 17:46:01.337254 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.337228    5315 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 17:46:01.337334 ip-10-0-128-204 kubenswrapper[5315]: E1103 17:46:01.337282    5315 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-204.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-204.us-west-2.compute.internal"
Nov 03 17:46:01.337334 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.337298    5315 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-204.us-west-2.compute.internal"
Nov 03 17:46:01.337412 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.337366    5315 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:46:01.346234 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.346212    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8a90b512115a10076b69387005e99589-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-204.us-west-2.compute.internal\" (UID: \"8a90b512115a10076b69387005e99589\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-204.us-west-2.compute.internal"
Nov 03 17:46:01.346234 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.346239    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a90b512115a10076b69387005e99589-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-204.us-west-2.compute.internal\" (UID: \"8a90b512115a10076b69387005e99589\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-204.us-west-2.compute.internal"
Nov 03 17:46:01.346365 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.346259    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8dbd524262c9b524921d913509ef36fa-config\") pod \"kube-apiserver-proxy-ip-10-0-128-204.us-west-2.compute.internal\" (UID: \"8dbd524262c9b524921d913509ef36fa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-204.us-west-2.compute.internal"
Nov 03 17:46:01.392527 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.392503    5315 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 17:46:01.392640 ip-10-0-128-204 kubenswrapper[5315]: E1103 17:46:01.392548    5315 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-204.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-204.us-west-2.compute.internal"
Nov 03 17:46:01.396029 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.396009    5315 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-128-204.us-west-2.compute.internal"
Nov 03 17:46:01.396098 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.396081    5315 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-204.us-west-2.compute.internal"
Nov 03 17:46:01.396666 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.396649    5315 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-204.us-west-2.compute.internal" event="NodeNotReady"
Nov 03 17:46:01.396748 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.396684    5315 setters.go:546] "Node became not ready" node="ip-10-0-128-204.us-west-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-03T17:46:01Z","lastTransitionTime":"2026-11-03T17:46:01Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Nov 03 17:46:01.445848 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.445758    5315 nodeinfomanager.go:471] existing CSINode "ip-10-0-128-204.us-west-2.compute.internal" is owned by different node (oldNodeID="6d8cc173-7a79-4c51-a19a-0278191d07de", newNodeID=""), cleaning up...
Nov 03 17:46:01.446372 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.446346    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8a90b512115a10076b69387005e99589-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-204.us-west-2.compute.internal\" (UID: \"8a90b512115a10076b69387005e99589\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-204.us-west-2.compute.internal"
Nov 03 17:46:01.446422 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.446389    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a90b512115a10076b69387005e99589-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-204.us-west-2.compute.internal\" (UID: \"8a90b512115a10076b69387005e99589\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-204.us-west-2.compute.internal"
Nov 03 17:46:01.446453 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.446421    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8dbd524262c9b524921d913509ef36fa-config\") pod \"kube-apiserver-proxy-ip-10-0-128-204.us-west-2.compute.internal\" (UID: \"8dbd524262c9b524921d913509ef36fa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-204.us-west-2.compute.internal"
Nov 03 17:46:01.446453 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.446444    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8a90b512115a10076b69387005e99589-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-204.us-west-2.compute.internal\" (UID: \"8a90b512115a10076b69387005e99589\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-204.us-west-2.compute.internal"
Nov 03 17:46:01.446511 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.446470    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8dbd524262c9b524921d913509ef36fa-config\") pod \"kube-apiserver-proxy-ip-10-0-128-204.us-west-2.compute.internal\" (UID: \"8dbd524262c9b524921d913509ef36fa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-204.us-west-2.compute.internal"
Nov 03 17:46:01.446540 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:01.446511    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a90b512115a10076b69387005e99589-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-204.us-west-2.compute.internal\" (UID: \"8a90b512115a10076b69387005e99589\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-204.us-west-2.compute.internal"
Nov 03 17:46:02.046427 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.046392    5315 apiserver.go:52] "Watching apiserver"
Nov 03 17:46:02.142914 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.142875    5315 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 17:46:02.143401 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.143380    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-4xjw2","openshift-monitoring/prometheus-operator-9cd89b74c-4wksp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6bks","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-204.us-west-2.compute.internal","openshift-monitoring/thanos-querier-74479fcb5-w59vr","openshift-multus/multus-jbbwp","openshift-network-operator/iptables-alerter-n9qsq","openshift-dns/node-resolver-9j547","openshift-multus/multus-additional-cni-plugins-bsgc5","openshift-dns/dns-default-pf6z7","openshift-insights/insights-runtime-extractor-qmcxl","openshift-monitoring/monitoring-plugin-77f4f76594-dwxs2","openshift-ingress-canary/ingress-canary-tfmb2","openshift-ovn-kubernetes/ovnkube-node-97p65","openshift-multus/network-metrics-daemon-z264t","openshift-network-diagnostics/network-check-target-2t57w","kube-system/kube-apiserver-proxy-ip-10-0-128-204.us-west-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-fjrg4","openshift-image-registry/node-ca-x8f2p","kube-system/konnectivity-agent-dv4zl","openshift-monitoring/node-exporter-7w9wb"]
Nov 03 17:46:02.151944 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.151914    5315 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-204.us-west-2.compute.internal"
Nov 03 17:46:02.152100 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.152043    5315 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-204.us-west-2.compute.internal"
Nov 03 17:46:02.161806 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.161779    5315 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 17:46:02.196969 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.196925    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 17:46:02.196969 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.196929    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-bmd8j\""
Nov 03 17:46:02.196969 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.196941    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8lnxv\""
Nov 03 17:46:02.197214 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.196987    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 17:46:02.197214 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.197028    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-sq9vz\""
Nov 03 17:46:02.197214 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.197028    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mk8dz\""
Nov 03 17:46:02.197214 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.197084    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5hxz7\""
Nov 03 17:46:02.197214 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.197109    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-j4vc9\""
Nov 03 17:46:02.197214 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.197117    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-z56rs\""
Nov 03 17:46:02.197397 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.197250    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-m6lsw\""
Nov 03 17:46:02.198036 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.198018    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-l7l2n\""
Nov 03 17:46:02.198036 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.198036    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6lzh9\""
Nov 03 17:46:02.198153 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.198115    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 17:46:02.198429 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.198414    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 17:46:02.198506 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.198491    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 17:46:02.198556 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.198525    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 17:46:02.198702 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.198686    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 17:46:02.198739 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.198704    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f85t9\""
Nov 03 17:46:02.198851 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.198836    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 17:46:02.198910 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.198875    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 17:46:02.199032 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.199016    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 17:46:02.199092 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.199032    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 17:46:02.199092 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.199046    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 17:46:02.199092 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.199058    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 17:46:02.199092 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.199041    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 17:46:02.199092 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.199088    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 17:46:02.199316 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.199095    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 17:46:02.199316 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.199103    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 17:46:02.199316 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.199116    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8w5gn\""
Nov 03 17:46:02.199316 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.199017    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 17:46:02.199316 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.199127    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 17:46:02.199316 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.199162    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 17:46:02.199316 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.199258    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 17:46:02.199316 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.199258    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 17:46:02.199316 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.199304    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 17:46:02.199557 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.199352    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 17:46:02.199557 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.199398    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 17:46:02.199557 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.199437    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 17:46:02.199557 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.199501    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 17:46:02.199557 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.199530    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 17:46:02.199557 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.199538    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 17:46:02.199725 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.199644    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 17:46:02.234721 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.234686    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 17:46:02.234899 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.234754    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 17:46:02.234899 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.234765    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 17:46:02.234899 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.234810    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 17:46:02.234899 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.234844    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nvxmw\""
Nov 03 17:46:02.235062 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.235018    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 17:46:02.235062 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.235020    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 17:46:02.235163 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.235108    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 17:46:02.235163 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.235111    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 17:46:02.237335 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.237320    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 17:46:02.237379 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.237359    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 03 17:46:02.237623 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.237598    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Nov 03 17:46:02.237786 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.237631    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xfh9g\""
Nov 03 17:46:02.237786 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.237601    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 17:46:02.237786 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.237674    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pvbgk\""
Nov 03 17:46:02.237786 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.237690    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-5q5nr\""
Nov 03 17:46:02.237786 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.237737    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 03 17:46:02.238644 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.238583    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 17:46:02.238733 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.238720    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 17:46:02.238851 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.238837    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 17:46:02.238921 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.238910    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-mtvgp\""
Nov 03 17:46:02.239441 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.239423    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 03 17:46:02.239670 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.239656    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 17:46:02.239707 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.239660    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 03 17:46:02.239892 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.239872    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 17:46:02.239892 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.239883    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 03 17:46:02.239892 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.239878    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 03 17:46:02.241599 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.241587    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 17:46:02.241599 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.241590    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-556wq\""
Nov 03 17:46:02.242786 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.242760    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-9v3v2ob5pcs9m\""
Nov 03 17:46:02.242786 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.242760    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Nov 03 17:46:02.242913 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.242773    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-xfqrm\""
Nov 03 17:46:02.245770 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.245755    5315 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 17:46:02.245821 ip-10-0-128-204 kubenswrapper[5315]: E1103 17:46:02.245806    5315 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-204.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-204.us-west-2.compute.internal"
Nov 03 17:46:02.246205 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.246193    5315 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 17:46:02.246252 ip-10-0-128-204 kubenswrapper[5315]: E1103 17:46:02.246227    5315 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-204.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-204.us-west-2.compute.internal"
Nov 03 17:46:02.251587 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.251563    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b993b313-db8b-42f5-ad99-2d2ab5bb49e0-system-cni-dir\") pod \"multus-additional-cni-plugins-bsgc5\" (UID: \"b993b313-db8b-42f5-ad99-2d2ab5bb49e0\") " pod="openshift-multus/multus-additional-cni-plugins-bsgc5"
Nov 03 17:46:02.251731 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.251595    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-etc-sysconfig\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:46:02.251731 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.251625    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cf65c17-1c79-4cd5-9ef1-dceb467878b0-metrics-certs\") pod \"network-metrics-daemon-z264t\" (UID: \"9cf65c17-1c79-4cd5-9ef1-dceb467878b0\") " pod="openshift-multus/network-metrics-daemon-z264t"
Nov 03 17:46:02.251731 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.251645    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30-dbus\") pod \"global-pull-secret-syncer-4xjw2\" (UID: \"f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30\") " pod="kube-system/global-pull-secret-syncer-4xjw2"
Nov 03 17:46:02.251731 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.251702    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-host-run-netns\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.251864 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.251848    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-etc-openvswitch\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.251904 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.251887    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b57a39bc-800c-4ea8-a0ef-4a1721786b70-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-4wksp\" (UID: \"b57a39bc-800c-4ea8-a0ef-4a1721786b70\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4wksp"
Nov 03 17:46:02.251955 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.251920    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-run\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:46:02.251955 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.251944    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-lib-modules\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:46:02.252041 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.251950    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cf65c17-1c79-4cd5-9ef1-dceb467878b0-metrics-certs\") pod \"network-metrics-daemon-z264t\" (UID: \"9cf65c17-1c79-4cd5-9ef1-dceb467878b0\") " pod="openshift-multus/network-metrics-daemon-z264t"
Nov 03 17:46:02.252041 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.251971    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/55b54081-644c-464e-91e1-15c2d8f5be59-node-exporter-accelerators-collector-config\") pod \"node-exporter-7w9wb\" (UID: \"55b54081-644c-464e-91e1-15c2d8f5be59\") " pod="openshift-monitoring/node-exporter-7w9wb"
Nov 03 17:46:02.252041 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.252007    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d59f146b-82ee-4f26-854a-cba274a89600-device-dir\") pod \"aws-ebs-csi-driver-node-v6bks\" (UID: \"d59f146b-82ee-4f26-854a-cba274a89600\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6bks"
Nov 03 17:46:02.252200 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.252098    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a831ae97-8f46-4b85-bfdb-59042b7b8b0c-hosts-file\") pod \"node-resolver-9j547\" (UID: \"a831ae97-8f46-4b85-bfdb-59042b7b8b0c\") " pod="openshift-dns/node-resolver-9j547"
Nov 03 17:46:02.252200 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.252142    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d59f146b-82ee-4f26-854a-cba274a89600-registration-dir\") pod \"aws-ebs-csi-driver-node-v6bks\" (UID: \"d59f146b-82ee-4f26-854a-cba274a89600\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6bks"
Nov 03 17:46:02.252200 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.252164    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b57a39bc-800c-4ea8-a0ef-4a1721786b70-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-4wksp\" (UID: \"b57a39bc-800c-4ea8-a0ef-4a1721786b70\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4wksp"
Nov 03 17:46:02.252200 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.252189    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-host-run-multus-certs\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:46:02.252390 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.252199    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/55b54081-644c-464e-91e1-15c2d8f5be59-node-exporter-accelerators-collector-config\") pod \"node-exporter-7w9wb\" (UID: \"55b54081-644c-464e-91e1-15c2d8f5be59\") " pod="openshift-monitoring/node-exporter-7w9wb"
Nov 03 17:46:02.252390 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.252237    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-log-socket\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.252390 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.252269    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e881e23-d118-4245-b855-b1322d133ef1-metrics-tls\") pod \"dns-default-pf6z7\" (UID: \"3e881e23-d118-4245-b855-b1322d133ef1\") " pod="openshift-dns/dns-default-pf6z7"
Nov 03 17:46:02.252390 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.252298    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/55b54081-644c-464e-91e1-15c2d8f5be59-node-exporter-tls\") pod \"node-exporter-7w9wb\" (UID: \"55b54081-644c-464e-91e1-15c2d8f5be59\") " pod="openshift-monitoring/node-exporter-7w9wb"
Nov 03 17:46:02.252390 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.252324    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/778383aa-b3db-45a6-8bec-9891f7818e94-host-slash\") pod \"iptables-alerter-n9qsq\" (UID: \"778383aa-b3db-45a6-8bec-9891f7818e94\") " pod="openshift-network-operator/iptables-alerter-n9qsq"
Nov 03 17:46:02.252390 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.252348    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d59f146b-82ee-4f26-854a-cba274a89600-etc-selinux\") pod \"aws-ebs-csi-driver-node-v6bks\" (UID: \"d59f146b-82ee-4f26-854a-cba274a89600\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6bks"
Nov 03 17:46:02.252665 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.252465    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-host-run-netns\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:46:02.252665 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.252492    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e0f2b93-0bed-447a-99ee-5b47f3bc09bf-cert\") pod \"ingress-canary-tfmb2\" (UID: \"6e0f2b93-0bed-447a-99ee-5b47f3bc09bf\") " pod="openshift-ingress-canary/ingress-canary-tfmb2"
Nov 03 17:46:02.252665 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.252500    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e881e23-d118-4245-b855-b1322d133ef1-metrics-tls\") pod \"dns-default-pf6z7\" (UID: \"3e881e23-d118-4245-b855-b1322d133ef1\") " pod="openshift-dns/dns-default-pf6z7"
Nov 03 17:46:02.252665 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.252517    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d505ffde-320b-41c5-a02e-374ee52e6a1a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-74479fcb5-w59vr\" (UID: \"d505ffde-320b-41c5-a02e-374ee52e6a1a\") " pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr"
Nov 03 17:46:02.252665 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.252525    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/55b54081-644c-464e-91e1-15c2d8f5be59-node-exporter-tls\") pod \"node-exporter-7w9wb\" (UID: \"55b54081-644c-464e-91e1-15c2d8f5be59\") " pod="openshift-monitoring/node-exporter-7w9wb"
Nov 03 17:46:02.252665 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.252547    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/55b54081-644c-464e-91e1-15c2d8f5be59-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7w9wb\" (UID: \"55b54081-644c-464e-91e1-15c2d8f5be59\") " pod="openshift-monitoring/node-exporter-7w9wb"
Nov 03 17:46:02.252665 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.252572    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-cnibin\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:46:02.252665 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.252597    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-multus-socket-dir-parent\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:46:02.252665 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.252641    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-node-log\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.252988 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.252671    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c29d200a-c917-46e5-9cf5-21f4d952e470-ovnkube-config\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.252988 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.252681    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e0f2b93-0bed-447a-99ee-5b47f3bc09bf-cert\") pod \"ingress-canary-tfmb2\" (UID: \"6e0f2b93-0bed-447a-99ee-5b47f3bc09bf\") " pod="openshift-ingress-canary/ingress-canary-tfmb2"
Nov 03 17:46:02.252988 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.252687    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d505ffde-320b-41c5-a02e-374ee52e6a1a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-74479fcb5-w59vr\" (UID: \"d505ffde-320b-41c5-a02e-374ee52e6a1a\") " pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr"
Nov 03 17:46:02.252988 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.252743    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-run-systemd\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.252988 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.252768    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-host-cni-bin\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.252988 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.252773    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/55b54081-644c-464e-91e1-15c2d8f5be59-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7w9wb\" (UID: \"55b54081-644c-464e-91e1-15c2d8f5be59\") " pod="openshift-monitoring/node-exporter-7w9wb"
Nov 03 17:46:02.252988 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.252786    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c29d200a-c917-46e5-9cf5-21f4d952e470-ovnkube-script-lib\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.252988 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.252822    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/778383aa-b3db-45a6-8bec-9891f7818e94-iptables-alerter-script\") pod \"iptables-alerter-n9qsq\" (UID: \"778383aa-b3db-45a6-8bec-9891f7818e94\") " pod="openshift-network-operator/iptables-alerter-n9qsq"
Nov 03 17:46:02.252988 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.252850    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zn5r\" (UniqueName: \"kubernetes.io/projected/b8a607f9-6133-4848-bdd6-d48e447cec86-kube-api-access-2zn5r\") pod \"insights-runtime-extractor-qmcxl\" (UID: \"b8a607f9-6133-4848-bdd6-d48e447cec86\") " pod="openshift-insights/insights-runtime-extractor-qmcxl"
Nov 03 17:46:02.252988 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.252874    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/55b54081-644c-464e-91e1-15c2d8f5be59-metrics-client-ca\") pod \"node-exporter-7w9wb\" (UID: \"55b54081-644c-464e-91e1-15c2d8f5be59\") " pod="openshift-monitoring/node-exporter-7w9wb"
Nov 03 17:46:02.252988 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.252890    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c29d200a-c917-46e5-9cf5-21f4d952e470-ovnkube-config\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.252988 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.252939    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c29d200a-c917-46e5-9cf5-21f4d952e470-ovnkube-script-lib\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.252988 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.252970    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b993b313-db8b-42f5-ad99-2d2ab5bb49e0-cni-binary-copy\") pod \"multus-additional-cni-plugins-bsgc5\" (UID: \"b993b313-db8b-42f5-ad99-2d2ab5bb49e0\") " pod="openshift-multus/multus-additional-cni-plugins-bsgc5"
Nov 03 17:46:02.253542 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.252999    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq9fz\" (UniqueName: \"kubernetes.io/projected/b993b313-db8b-42f5-ad99-2d2ab5bb49e0-kube-api-access-kq9fz\") pod \"multus-additional-cni-plugins-bsgc5\" (UID: \"b993b313-db8b-42f5-ad99-2d2ab5bb49e0\") " pod="openshift-multus/multus-additional-cni-plugins-bsgc5"
Nov 03 17:46:02.253542 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.253024    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-etc-sysctl-d\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:46:02.253542 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.253048    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d59f146b-82ee-4f26-854a-cba274a89600-socket-dir\") pod \"aws-ebs-csi-driver-node-v6bks\" (UID: \"d59f146b-82ee-4f26-854a-cba274a89600\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6bks"
Nov 03 17:46:02.253542 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.253058    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/55b54081-644c-464e-91e1-15c2d8f5be59-metrics-client-ca\") pod \"node-exporter-7w9wb\" (UID: \"55b54081-644c-464e-91e1-15c2d8f5be59\") " pod="openshift-monitoring/node-exporter-7w9wb"
Nov 03 17:46:02.253542 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.253071    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-multus-conf-dir\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:46:02.253542 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.253074    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/778383aa-b3db-45a6-8bec-9891f7818e94-iptables-alerter-script\") pod \"iptables-alerter-n9qsq\" (UID: \"778383aa-b3db-45a6-8bec-9891f7818e94\") " pod="openshift-network-operator/iptables-alerter-n9qsq"
Nov 03 17:46:02.253542 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.253094    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-host-run-ovn-kubernetes\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.253542 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.253101    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b993b313-db8b-42f5-ad99-2d2ab5bb49e0-cni-binary-copy\") pod \"multus-additional-cni-plugins-bsgc5\" (UID: \"b993b313-db8b-42f5-ad99-2d2ab5bb49e0\") " pod="openshift-multus/multus-additional-cni-plugins-bsgc5"
Nov 03 17:46:02.253542 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.253136    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79212f04-d10c-43ee-9dd5-41b4fd1917b2-host\") pod \"node-ca-x8f2p\" (UID: \"79212f04-d10c-43ee-9dd5-41b4fd1917b2\") " pod="openshift-image-registry/node-ca-x8f2p"
Nov 03 17:46:02.253542 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.253183    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtbgb\" (UniqueName: \"kubernetes.io/projected/9cf65c17-1c79-4cd5-9ef1-dceb467878b0-kube-api-access-vtbgb\") pod \"network-metrics-daemon-z264t\" (UID: \"9cf65c17-1c79-4cd5-9ef1-dceb467878b0\") " pod="openshift-multus/network-metrics-daemon-z264t"
Nov 03 17:46:02.253542 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.253217    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a831ae97-8f46-4b85-bfdb-59042b7b8b0c-tmp-dir\") pod \"node-resolver-9j547\" (UID: \"a831ae97-8f46-4b85-bfdb-59042b7b8b0c\") " pod="openshift-dns/node-resolver-9j547"
Nov 03 17:46:02.253542 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.253242    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d505ffde-320b-41c5-a02e-374ee52e6a1a-metrics-client-ca\") pod \"thanos-querier-74479fcb5-w59vr\" (UID: \"d505ffde-320b-41c5-a02e-374ee52e6a1a\") " pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr"
Nov 03 17:46:02.253542 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.253268    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-sys\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:46:02.253542 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.253283    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/55b54081-644c-464e-91e1-15c2d8f5be59-sys\") pod \"node-exporter-7w9wb\" (UID: \"55b54081-644c-464e-91e1-15c2d8f5be59\") " pod="openshift-monitoring/node-exporter-7w9wb"
Nov 03 17:46:02.253542 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.253300    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfmqk\" (UniqueName: \"kubernetes.io/projected/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-kube-api-access-rfmqk\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:46:02.253542 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.253314    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a831ae97-8f46-4b85-bfdb-59042b7b8b0c-tmp-dir\") pod \"node-resolver-9j547\" (UID: \"a831ae97-8f46-4b85-bfdb-59042b7b8b0c\") " pod="openshift-dns/node-resolver-9j547"
Nov 03 17:46:02.253542 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.253320    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/55b54081-644c-464e-91e1-15c2d8f5be59-root\") pod \"node-exporter-7w9wb\" (UID: \"55b54081-644c-464e-91e1-15c2d8f5be59\") " pod="openshift-monitoring/node-exporter-7w9wb"
Nov 03 17:46:02.253542 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.253373    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jstr\" (UniqueName: \"kubernetes.io/projected/55b54081-644c-464e-91e1-15c2d8f5be59-kube-api-access-2jstr\") pod \"node-exporter-7w9wb\" (UID: \"55b54081-644c-464e-91e1-15c2d8f5be59\") " pod="openshift-monitoring/node-exporter-7w9wb"
Nov 03 17:46:02.253542 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.253417    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d505ffde-320b-41c5-a02e-374ee52e6a1a-metrics-client-ca\") pod \"thanos-querier-74479fcb5-w59vr\" (UID: \"d505ffde-320b-41c5-a02e-374ee52e6a1a\") " pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr"
Nov 03 17:46:02.253542 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.253482    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d59f146b-82ee-4f26-854a-cba274a89600-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v6bks\" (UID: \"d59f146b-82ee-4f26-854a-cba274a89600\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6bks"
Nov 03 17:46:02.254506 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.253582    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b57a39bc-800c-4ea8-a0ef-4a1721786b70-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-4wksp\" (UID: \"b57a39bc-800c-4ea8-a0ef-4a1721786b70\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4wksp"
Nov 03 17:46:02.254506 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.253625    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d505ffde-320b-41c5-a02e-374ee52e6a1a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-74479fcb5-w59vr\" (UID: \"d505ffde-320b-41c5-a02e-374ee52e6a1a\") " pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr"
Nov 03 17:46:02.254506 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.253651    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d505ffde-320b-41c5-a02e-374ee52e6a1a-tmp\") pod \"thanos-querier-74479fcb5-w59vr\" (UID: \"d505ffde-320b-41c5-a02e-374ee52e6a1a\") " pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr"
Nov 03 17:46:02.254506 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.253674    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b993b313-db8b-42f5-ad99-2d2ab5bb49e0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bsgc5\" (UID: \"b993b313-db8b-42f5-ad99-2d2ab5bb49e0\") " pod="openshift-multus/multus-additional-cni-plugins-bsgc5"
Nov 03 17:46:02.254506 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.253691    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b993b313-db8b-42f5-ad99-2d2ab5bb49e0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bsgc5\" (UID: \"b993b313-db8b-42f5-ad99-2d2ab5bb49e0\") " pod="openshift-multus/multus-additional-cni-plugins-bsgc5"
Nov 03 17:46:02.254506 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.253712    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-host\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:46:02.254506 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.253727    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-multus-daemon-config\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:46:02.254506 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.253733    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b57a39bc-800c-4ea8-a0ef-4a1721786b70-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-4wksp\" (UID: \"b57a39bc-800c-4ea8-a0ef-4a1721786b70\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4wksp"
Nov 03 17:46:02.254506 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.253744    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-var-lib-openvswitch\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.254506 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.253761    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jssl5\" (UniqueName: \"kubernetes.io/projected/c29d200a-c917-46e5-9cf5-21f4d952e470-kube-api-access-jssl5\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.254506 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.253767    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d505ffde-320b-41c5-a02e-374ee52e6a1a-tmp\") pod \"thanos-querier-74479fcb5-w59vr\" (UID: \"d505ffde-320b-41c5-a02e-374ee52e6a1a\") " pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr"
Nov 03 17:46:02.254506 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.253783    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-etc-systemd\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:46:02.254506 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.253816    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b91c00dc-c14f-4b3a-b268-1eb6dcf9ad4b-agent-certs\") pod \"konnectivity-agent-dv4zl\" (UID: \"b91c00dc-c14f-4b3a-b268-1eb6dcf9ad4b\") " pod="kube-system/konnectivity-agent-dv4zl"
Nov 03 17:46:02.254506 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.253869    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b993b313-db8b-42f5-ad99-2d2ab5bb49e0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bsgc5\" (UID: \"b993b313-db8b-42f5-ad99-2d2ab5bb49e0\") " pod="openshift-multus/multus-additional-cni-plugins-bsgc5"
Nov 03 17:46:02.254506 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.253880    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d505ffde-320b-41c5-a02e-374ee52e6a1a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-74479fcb5-w59vr\" (UID: \"d505ffde-320b-41c5-a02e-374ee52e6a1a\") " pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr"
Nov 03 17:46:02.254506 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.253925    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b993b313-db8b-42f5-ad99-2d2ab5bb49e0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bsgc5\" (UID: \"b993b313-db8b-42f5-ad99-2d2ab5bb49e0\") " pod="openshift-multus/multus-additional-cni-plugins-bsgc5"
Nov 03 17:46:02.254506 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.253925    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b91c00dc-c14f-4b3a-b268-1eb6dcf9ad4b-konnectivity-ca\") pod \"konnectivity-agent-dv4zl\" (UID: \"b91c00dc-c14f-4b3a-b268-1eb6dcf9ad4b\") " pod="kube-system/konnectivity-agent-dv4zl"
Nov 03 17:46:02.254506 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.254033    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-multus-daemon-config\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:46:02.254506 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.254037    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-host-run-k8s-cni-cncf-io\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:46:02.254506 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.254060    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/79212f04-d10c-43ee-9dd5-41b4fd1917b2-serviceca\") pod \"node-ca-x8f2p\" (UID: \"79212f04-d10c-43ee-9dd5-41b4fd1917b2\") " pod="openshift-image-registry/node-ca-x8f2p"
Nov 03 17:46:02.254506 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.254069    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b91c00dc-c14f-4b3a-b268-1eb6dcf9ad4b-agent-certs\") pod \"konnectivity-agent-dv4zl\" (UID: \"b91c00dc-c14f-4b3a-b268-1eb6dcf9ad4b\") " pod="kube-system/konnectivity-agent-dv4zl"
Nov 03 17:46:02.254506 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.254078    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b8a607f9-6133-4848-bdd6-d48e447cec86-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qmcxl\" (UID: \"b8a607f9-6133-4848-bdd6-d48e447cec86\") " pod="openshift-insights/insights-runtime-extractor-qmcxl"
Nov 03 17:46:02.254506 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.254110    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c29d200a-c917-46e5-9cf5-21f4d952e470-ovn-node-metrics-cert\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.254506 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.254147    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b91c00dc-c14f-4b3a-b268-1eb6dcf9ad4b-konnectivity-ca\") pod \"konnectivity-agent-dv4zl\" (UID: \"b91c00dc-c14f-4b3a-b268-1eb6dcf9ad4b\") " pod="kube-system/konnectivity-agent-dv4zl"
Nov 03 17:46:02.254506 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.254186    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-etc-sysctl-conf\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:46:02.254506 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.254228    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-host-kubelet\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.254506 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.254263    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-host-slash\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.254506 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.254268    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b8a607f9-6133-4848-bdd6-d48e447cec86-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qmcxl\" (UID: \"b8a607f9-6133-4848-bdd6-d48e447cec86\") " pod="openshift-insights/insights-runtime-extractor-qmcxl"
Nov 03 17:46:02.254506 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.254284    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c29d200a-c917-46e5-9cf5-21f4d952e470-env-overrides\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.254506 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.254417    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c29d200a-c917-46e5-9cf5-21f4d952e470-ovn-node-metrics-cert\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.254506 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.254437    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c29d200a-c917-46e5-9cf5-21f4d952e470-env-overrides\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.254506 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.254454    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b57a39bc-800c-4ea8-a0ef-4a1721786b70-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-4wksp\" (UID: \"b57a39bc-800c-4ea8-a0ef-4a1721786b70\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4wksp"
Nov 03 17:46:02.254506 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.254484    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt25m\" (UniqueName: \"kubernetes.io/projected/778383aa-b3db-45a6-8bec-9891f7818e94-kube-api-access-dt25m\") pod \"iptables-alerter-n9qsq\" (UID: \"778383aa-b3db-45a6-8bec-9891f7818e94\") " pod="openshift-network-operator/iptables-alerter-n9qsq"
Nov 03 17:46:02.254506 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.254512    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jsj9\" (UniqueName: \"kubernetes.io/projected/3e881e23-d118-4245-b855-b1322d133ef1-kube-api-access-9jsj9\") pod \"dns-default-pf6z7\" (UID: \"3e881e23-d118-4245-b855-b1322d133ef1\") " pod="openshift-dns/dns-default-pf6z7"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.254543    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-host-var-lib-cni-bin\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.254565    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-etc-kubernetes\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.254588    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-system-cni-dir\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.254628    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-systemd-units\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.254652    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-etc-modprobe-d\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.254675    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/55b54081-644c-464e-91e1-15c2d8f5be59-node-exporter-wtmp\") pod \"node-exporter-7w9wb\" (UID: \"55b54081-644c-464e-91e1-15c2d8f5be59\") " pod="openshift-monitoring/node-exporter-7w9wb"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.254698    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-host-cni-netd\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.254727    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.254747    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b57a39bc-800c-4ea8-a0ef-4a1721786b70-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-4wksp\" (UID: \"b57a39bc-800c-4ea8-a0ef-4a1721786b70\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4wksp"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.254754    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e106ef93-623a-4cb4-9848-31087f1a7711-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-dwxs2\" (UID: \"e106ef93-623a-4cb4-9848-31087f1a7711\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-dwxs2"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.254783    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8a607f9-6133-4848-bdd6-d48e447cec86-data-volume\") pod \"insights-runtime-extractor-qmcxl\" (UID: \"b8a607f9-6133-4848-bdd6-d48e447cec86\") " pod="openshift-insights/insights-runtime-extractor-qmcxl"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.254806    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/55b54081-644c-464e-91e1-15c2d8f5be59-node-exporter-textfile\") pod \"node-exporter-7w9wb\" (UID: \"55b54081-644c-464e-91e1-15c2d8f5be59\") " pod="openshift-monitoring/node-exporter-7w9wb"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.254833    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30-kubelet-config\") pod \"global-pull-secret-syncer-4xjw2\" (UID: \"f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30\") " pod="kube-system/global-pull-secret-syncer-4xjw2"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.254860    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-multus-cni-dir\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.254883    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-host-var-lib-cni-multus\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.254908    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-var-lib-kubelet\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.254933    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-run-openvswitch\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.254960    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b8a607f9-6133-4848-bdd6-d48e447cec86-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qmcxl\" (UID: \"b8a607f9-6133-4848-bdd6-d48e447cec86\") " pod="openshift-insights/insights-runtime-extractor-qmcxl"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.254962    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/79212f04-d10c-43ee-9dd5-41b4fd1917b2-serviceca\") pod \"node-ca-x8f2p\" (UID: \"79212f04-d10c-43ee-9dd5-41b4fd1917b2\") " pod="openshift-image-registry/node-ca-x8f2p"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.254998    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjnd5\" (UniqueName: \"kubernetes.io/projected/b57a39bc-800c-4ea8-a0ef-4a1721786b70-kube-api-access-fjnd5\") pod \"prometheus-operator-9cd89b74c-4wksp\" (UID: \"b57a39bc-800c-4ea8-a0ef-4a1721786b70\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4wksp"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.255040    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d505ffde-320b-41c5-a02e-374ee52e6a1a-secret-thanos-querier-tls\") pod \"thanos-querier-74479fcb5-w59vr\" (UID: \"d505ffde-320b-41c5-a02e-374ee52e6a1a\") " pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.255047    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8a607f9-6133-4848-bdd6-d48e447cec86-data-volume\") pod \"insights-runtime-extractor-qmcxl\" (UID: \"b8a607f9-6133-4848-bdd6-d48e447cec86\") " pod="openshift-insights/insights-runtime-extractor-qmcxl"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.255067    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d505ffde-320b-41c5-a02e-374ee52e6a1a-secret-grpc-tls\") pod \"thanos-querier-74479fcb5-w59vr\" (UID: \"d505ffde-320b-41c5-a02e-374ee52e6a1a\") " pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.255107    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njbgq\" (UniqueName: \"kubernetes.io/projected/d505ffde-320b-41c5-a02e-374ee52e6a1a-kube-api-access-njbgq\") pod \"thanos-querier-74479fcb5-w59vr\" (UID: \"d505ffde-320b-41c5-a02e-374ee52e6a1a\") " pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.255211    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3e881e23-d118-4245-b855-b1322d133ef1-config-volume\") pod \"dns-default-pf6z7\" (UID: \"3e881e23-d118-4245-b855-b1322d133ef1\") " pod="openshift-dns/dns-default-pf6z7"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.255242    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d505ffde-320b-41c5-a02e-374ee52e6a1a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-74479fcb5-w59vr\" (UID: \"d505ffde-320b-41c5-a02e-374ee52e6a1a\") " pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.255271    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-etc-kubernetes\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.255302    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v89s6\" (UniqueName: \"kubernetes.io/projected/43c9da28-334d-4bb3-a6bc-e3da2d561633-kube-api-access-v89s6\") pod \"network-check-target-2t57w\" (UID: \"43c9da28-334d-4bb3-a6bc-e3da2d561633\") " pod="openshift-network-diagnostics/network-check-target-2t57w"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.255329    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-hostroot\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.255341    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e106ef93-623a-4cb4-9848-31087f1a7711-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-dwxs2\" (UID: \"e106ef93-623a-4cb4-9848-31087f1a7711\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-dwxs2"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.255356    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d505ffde-320b-41c5-a02e-374ee52e6a1a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-74479fcb5-w59vr\" (UID: \"d505ffde-320b-41c5-a02e-374ee52e6a1a\") " pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.255365    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d505ffde-320b-41c5-a02e-374ee52e6a1a-secret-thanos-querier-tls\") pod \"thanos-querier-74479fcb5-w59vr\" (UID: \"d505ffde-320b-41c5-a02e-374ee52e6a1a\") " pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.255381    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b8a607f9-6133-4848-bdd6-d48e447cec86-crio-socket\") pod \"insights-runtime-extractor-qmcxl\" (UID: \"b8a607f9-6133-4848-bdd6-d48e447cec86\") " pod="openshift-insights/insights-runtime-extractor-qmcxl"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.255411    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-run-ovn\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.255441    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b993b313-db8b-42f5-ad99-2d2ab5bb49e0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bsgc5\" (UID: \"b993b313-db8b-42f5-ad99-2d2ab5bb49e0\") " pod="openshift-multus/multus-additional-cni-plugins-bsgc5"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.255470    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djdsz\" (UniqueName: \"kubernetes.io/projected/79212f04-d10c-43ee-9dd5-41b4fd1917b2-kube-api-access-djdsz\") pod \"node-ca-x8f2p\" (UID: \"79212f04-d10c-43ee-9dd5-41b4fd1917b2\") " pod="openshift-image-registry/node-ca-x8f2p"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.255497    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e881e23-d118-4245-b855-b1322d133ef1-tmp-dir\") pod \"dns-default-pf6z7\" (UID: \"3e881e23-d118-4245-b855-b1322d133ef1\") " pod="openshift-dns/dns-default-pf6z7"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.255565    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d505ffde-320b-41c5-a02e-374ee52e6a1a-secret-grpc-tls\") pod \"thanos-querier-74479fcb5-w59vr\" (UID: \"d505ffde-320b-41c5-a02e-374ee52e6a1a\") " pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.255687    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-tmp\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.255725    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpp49\" (UniqueName: \"kubernetes.io/projected/a831ae97-8f46-4b85-bfdb-59042b7b8b0c-kube-api-access-mpp49\") pod \"node-resolver-9j547\" (UID: \"a831ae97-8f46-4b85-bfdb-59042b7b8b0c\") " pod="openshift-dns/node-resolver-9j547"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.255742    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e881e23-d118-4245-b855-b1322d133ef1-tmp-dir\") pod \"dns-default-pf6z7\" (UID: \"3e881e23-d118-4245-b855-b1322d133ef1\") " pod="openshift-dns/dns-default-pf6z7"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.255691    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/55b54081-644c-464e-91e1-15c2d8f5be59-node-exporter-textfile\") pod \"node-exporter-7w9wb\" (UID: \"55b54081-644c-464e-91e1-15c2d8f5be59\") " pod="openshift-monitoring/node-exporter-7w9wb"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.255791    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b8a607f9-6133-4848-bdd6-d48e447cec86-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qmcxl\" (UID: \"b8a607f9-6133-4848-bdd6-d48e447cec86\") " pod="openshift-insights/insights-runtime-extractor-qmcxl"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.255793    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30-original-pull-secret\") pod \"global-pull-secret-syncer-4xjw2\" (UID: \"f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30\") " pod="kube-system/global-pull-secret-syncer-4xjw2"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.255808    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-tmp\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.255819    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3e881e23-d118-4245-b855-b1322d133ef1-config-volume\") pod \"dns-default-pf6z7\" (UID: \"3e881e23-d118-4245-b855-b1322d133ef1\") " pod="openshift-dns/dns-default-pf6z7"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.255829    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-cni-binary-copy\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.255850    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d505ffde-320b-41c5-a02e-374ee52e6a1a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-74479fcb5-w59vr\" (UID: \"d505ffde-320b-41c5-a02e-374ee52e6a1a\") " pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.255858    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cztkv\" (UniqueName: \"kubernetes.io/projected/e106ef93-623a-4cb4-9848-31087f1a7711-kube-api-access-cztkv\") pod \"monitoring-plugin-77f4f76594-dwxs2\" (UID: \"e106ef93-623a-4cb4-9848-31087f1a7711\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-dwxs2"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.255913    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-etc-tuned\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.255957    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d505ffde-320b-41c5-a02e-374ee52e6a1a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-74479fcb5-w59vr\" (UID: \"d505ffde-320b-41c5-a02e-374ee52e6a1a\") " pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.256013    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-etc-tuned\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:46:02.256037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.256050    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzjx9\" (UniqueName: \"kubernetes.io/projected/d59f146b-82ee-4f26-854a-cba274a89600-kube-api-access-bzjx9\") pod \"aws-ebs-csi-driver-node-v6bks\" (UID: \"d59f146b-82ee-4f26-854a-cba274a89600\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6bks"
Nov 03 17:46:02.257910 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.256077    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30-original-pull-secret\") pod \"global-pull-secret-syncer-4xjw2\" (UID: \"f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30\") " pod="kube-system/global-pull-secret-syncer-4xjw2"
Nov 03 17:46:02.257910 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.256085    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9tww\" (UniqueName: \"kubernetes.io/projected/6e0f2b93-0bed-447a-99ee-5b47f3bc09bf-kube-api-access-r9tww\") pod \"ingress-canary-tfmb2\" (UID: \"6e0f2b93-0bed-447a-99ee-5b47f3bc09bf\") " pod="openshift-ingress-canary/ingress-canary-tfmb2"
Nov 03 17:46:02.257910 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.256111    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-cni-binary-copy\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:46:02.257910 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.256172    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b993b313-db8b-42f5-ad99-2d2ab5bb49e0-cnibin\") pod \"multus-additional-cni-plugins-bsgc5\" (UID: \"b993b313-db8b-42f5-ad99-2d2ab5bb49e0\") " pod="openshift-multus/multus-additional-cni-plugins-bsgc5"
Nov 03 17:46:02.257910 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.256287    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d59f146b-82ee-4f26-854a-cba274a89600-sys-fs\") pod \"aws-ebs-csi-driver-node-v6bks\" (UID: \"d59f146b-82ee-4f26-854a-cba274a89600\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6bks"
Nov 03 17:46:02.257910 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.256342    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-host-var-lib-kubelet\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:46:02.257910 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.256383    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plbfz\" (UniqueName: \"kubernetes.io/projected/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-kube-api-access-plbfz\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:46:02.309250 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.309159    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfmqk\" (UniqueName: \"kubernetes.io/projected/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-kube-api-access-rfmqk\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:46:02.309415 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.309262    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zn5r\" (UniqueName: \"kubernetes.io/projected/b8a607f9-6133-4848-bdd6-d48e447cec86-kube-api-access-2zn5r\") pod \"insights-runtime-extractor-qmcxl\" (UID: \"b8a607f9-6133-4848-bdd6-d48e447cec86\") " pod="openshift-insights/insights-runtime-extractor-qmcxl"
Nov 03 17:46:02.309415 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.309315    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq9fz\" (UniqueName: \"kubernetes.io/projected/b993b313-db8b-42f5-ad99-2d2ab5bb49e0-kube-api-access-kq9fz\") pod \"multus-additional-cni-plugins-bsgc5\" (UID: \"b993b313-db8b-42f5-ad99-2d2ab5bb49e0\") " pod="openshift-multus/multus-additional-cni-plugins-bsgc5"
Nov 03 17:46:02.309549 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.309413    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jssl5\" (UniqueName: \"kubernetes.io/projected/c29d200a-c917-46e5-9cf5-21f4d952e470-kube-api-access-jssl5\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.309600 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.309570    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jstr\" (UniqueName: \"kubernetes.io/projected/55b54081-644c-464e-91e1-15c2d8f5be59-kube-api-access-2jstr\") pod \"node-exporter-7w9wb\" (UID: \"55b54081-644c-464e-91e1-15c2d8f5be59\") " pod="openshift-monitoring/node-exporter-7w9wb"
Nov 03 17:46:02.311842 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.311821    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt25m\" (UniqueName: \"kubernetes.io/projected/778383aa-b3db-45a6-8bec-9891f7818e94-kube-api-access-dt25m\") pod \"iptables-alerter-n9qsq\" (UID: \"778383aa-b3db-45a6-8bec-9891f7818e94\") " pod="openshift-network-operator/iptables-alerter-n9qsq"
Nov 03 17:46:02.311937 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.311873    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v89s6\" (UniqueName: \"kubernetes.io/projected/43c9da28-334d-4bb3-a6bc-e3da2d561633-kube-api-access-v89s6\") pod \"network-check-target-2t57w\" (UID: \"43c9da28-334d-4bb3-a6bc-e3da2d561633\") " pod="openshift-network-diagnostics/network-check-target-2t57w"
Nov 03 17:46:02.312838 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.312818    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jsj9\" (UniqueName: \"kubernetes.io/projected/3e881e23-d118-4245-b855-b1322d133ef1-kube-api-access-9jsj9\") pod \"dns-default-pf6z7\" (UID: \"3e881e23-d118-4245-b855-b1322d133ef1\") " pod="openshift-dns/dns-default-pf6z7"
Nov 03 17:46:02.312919 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.312902    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjnd5\" (UniqueName: \"kubernetes.io/projected/b57a39bc-800c-4ea8-a0ef-4a1721786b70-kube-api-access-fjnd5\") pod \"prometheus-operator-9cd89b74c-4wksp\" (UID: \"b57a39bc-800c-4ea8-a0ef-4a1721786b70\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4wksp"
Nov 03 17:46:02.313346 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.313326    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzjx9\" (UniqueName: \"kubernetes.io/projected/d59f146b-82ee-4f26-854a-cba274a89600-kube-api-access-bzjx9\") pod \"aws-ebs-csi-driver-node-v6bks\" (UID: \"d59f146b-82ee-4f26-854a-cba274a89600\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6bks"
Nov 03 17:46:02.313503 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.313485    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpp49\" (UniqueName: \"kubernetes.io/projected/a831ae97-8f46-4b85-bfdb-59042b7b8b0c-kube-api-access-mpp49\") pod \"node-resolver-9j547\" (UID: \"a831ae97-8f46-4b85-bfdb-59042b7b8b0c\") " pod="openshift-dns/node-resolver-9j547"
Nov 03 17:46:02.314404 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.314380    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtbgb\" (UniqueName: \"kubernetes.io/projected/9cf65c17-1c79-4cd5-9ef1-dceb467878b0-kube-api-access-vtbgb\") pod \"network-metrics-daemon-z264t\" (UID: \"9cf65c17-1c79-4cd5-9ef1-dceb467878b0\") " pod="openshift-multus/network-metrics-daemon-z264t"
Nov 03 17:46:02.314531 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.314510    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plbfz\" (UniqueName: \"kubernetes.io/projected/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-kube-api-access-plbfz\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:46:02.314565 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.314524    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9tww\" (UniqueName: \"kubernetes.io/projected/6e0f2b93-0bed-447a-99ee-5b47f3bc09bf-kube-api-access-r9tww\") pod \"ingress-canary-tfmb2\" (UID: \"6e0f2b93-0bed-447a-99ee-5b47f3bc09bf\") " pod="openshift-ingress-canary/ingress-canary-tfmb2"
Nov 03 17:46:02.314806 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.314790    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cztkv\" (UniqueName: \"kubernetes.io/projected/e106ef93-623a-4cb4-9848-31087f1a7711-kube-api-access-cztkv\") pod \"monitoring-plugin-77f4f76594-dwxs2\" (UID: \"e106ef93-623a-4cb4-9848-31087f1a7711\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-dwxs2"
Nov 03 17:46:02.319656 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.319636    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djdsz\" (UniqueName: \"kubernetes.io/projected/79212f04-d10c-43ee-9dd5-41b4fd1917b2-kube-api-access-djdsz\") pod \"node-ca-x8f2p\" (UID: \"79212f04-d10c-43ee-9dd5-41b4fd1917b2\") " pod="openshift-image-registry/node-ca-x8f2p"
Nov 03 17:46:02.319772 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.319756    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njbgq\" (UniqueName: \"kubernetes.io/projected/d505ffde-320b-41c5-a02e-374ee52e6a1a-kube-api-access-njbgq\") pod \"thanos-querier-74479fcb5-w59vr\" (UID: \"d505ffde-320b-41c5-a02e-374ee52e6a1a\") " pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr"
Nov 03 17:46:02.357935 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.357901    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-host-slash\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.357935 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.357936    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-host-var-lib-cni-bin\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:46:02.358119 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.357961    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-etc-kubernetes\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:46:02.358119 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.357975    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-system-cni-dir\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:46:02.358119 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358013    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-etc-kubernetes\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:46:02.358119 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358020    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-host-slash\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.358119 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358046    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-systemd-units\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.358119 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358020    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-host-var-lib-cni-bin\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:46:02.358119 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358077    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-etc-modprobe-d\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:46:02.358119 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358097    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/55b54081-644c-464e-91e1-15c2d8f5be59-node-exporter-wtmp\") pod \"node-exporter-7w9wb\" (UID: \"55b54081-644c-464e-91e1-15c2d8f5be59\") " pod="openshift-monitoring/node-exporter-7w9wb"
Nov 03 17:46:02.358119 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358112    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-host-cni-netd\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.358119 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358114    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-systemd-units\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.358497 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358120    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-system-cni-dir\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:46:02.358497 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358135    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.358497 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358166    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-host-cni-netd\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.358497 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358172    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.358497 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358175    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30-kubelet-config\") pod \"global-pull-secret-syncer-4xjw2\" (UID: \"f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30\") " pod="kube-system/global-pull-secret-syncer-4xjw2"
Nov 03 17:46:02.358497 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358198    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-etc-modprobe-d\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:46:02.358497 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358213    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-multus-cni-dir\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:46:02.358497 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358223    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30-kubelet-config\") pod \"global-pull-secret-syncer-4xjw2\" (UID: \"f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30\") " pod="kube-system/global-pull-secret-syncer-4xjw2"
Nov 03 17:46:02.358497 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358240    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-host-var-lib-cni-multus\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:46:02.358497 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358265    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-multus-cni-dir\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:46:02.358497 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358265    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-var-lib-kubelet\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:46:02.358497 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358295    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-host-var-lib-cni-multus\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:46:02.358497 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358305    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-var-lib-kubelet\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:46:02.358497 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358297    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-run-openvswitch\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.358497 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358327    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-run-openvswitch\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.358497 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358217    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/55b54081-644c-464e-91e1-15c2d8f5be59-node-exporter-wtmp\") pod \"node-exporter-7w9wb\" (UID: \"55b54081-644c-464e-91e1-15c2d8f5be59\") " pod="openshift-monitoring/node-exporter-7w9wb"
Nov 03 17:46:02.358497 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358352    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-etc-kubernetes\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:46:02.358497 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358371    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-hostroot\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:46:02.358497 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358397    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-hostroot\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:46:02.358497 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358403    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-etc-kubernetes\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:46:02.358497 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358435    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b8a607f9-6133-4848-bdd6-d48e447cec86-crio-socket\") pod \"insights-runtime-extractor-qmcxl\" (UID: \"b8a607f9-6133-4848-bdd6-d48e447cec86\") " pod="openshift-insights/insights-runtime-extractor-qmcxl"
Nov 03 17:46:02.358497 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358447    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b8a607f9-6133-4848-bdd6-d48e447cec86-crio-socket\") pod \"insights-runtime-extractor-qmcxl\" (UID: \"b8a607f9-6133-4848-bdd6-d48e447cec86\") " pod="openshift-insights/insights-runtime-extractor-qmcxl"
Nov 03 17:46:02.358497 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358463    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-run-ovn\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.358497 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358480    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b993b313-db8b-42f5-ad99-2d2ab5bb49e0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bsgc5\" (UID: \"b993b313-db8b-42f5-ad99-2d2ab5bb49e0\") " pod="openshift-multus/multus-additional-cni-plugins-bsgc5"
Nov 03 17:46:02.358497 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358506    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b993b313-db8b-42f5-ad99-2d2ab5bb49e0-cnibin\") pod \"multus-additional-cni-plugins-bsgc5\" (UID: \"b993b313-db8b-42f5-ad99-2d2ab5bb49e0\") " pod="openshift-multus/multus-additional-cni-plugins-bsgc5"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358524    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d59f146b-82ee-4f26-854a-cba274a89600-sys-fs\") pod \"aws-ebs-csi-driver-node-v6bks\" (UID: \"d59f146b-82ee-4f26-854a-cba274a89600\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6bks"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358538    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-host-var-lib-kubelet\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358559    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b993b313-db8b-42f5-ad99-2d2ab5bb49e0-cnibin\") pod \"multus-additional-cni-plugins-bsgc5\" (UID: \"b993b313-db8b-42f5-ad99-2d2ab5bb49e0\") " pod="openshift-multus/multus-additional-cni-plugins-bsgc5"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358559    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b993b313-db8b-42f5-ad99-2d2ab5bb49e0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bsgc5\" (UID: \"b993b313-db8b-42f5-ad99-2d2ab5bb49e0\") " pod="openshift-multus/multus-additional-cni-plugins-bsgc5"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358576    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-host-var-lib-kubelet\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358578    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-run-ovn\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358588    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d59f146b-82ee-4f26-854a-cba274a89600-sys-fs\") pod \"aws-ebs-csi-driver-node-v6bks\" (UID: \"d59f146b-82ee-4f26-854a-cba274a89600\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6bks"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358597    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b993b313-db8b-42f5-ad99-2d2ab5bb49e0-system-cni-dir\") pod \"multus-additional-cni-plugins-bsgc5\" (UID: \"b993b313-db8b-42f5-ad99-2d2ab5bb49e0\") " pod="openshift-multus/multus-additional-cni-plugins-bsgc5"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358639    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-etc-sysconfig\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358654    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30-dbus\") pod \"global-pull-secret-syncer-4xjw2\" (UID: \"f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30\") " pod="kube-system/global-pull-secret-syncer-4xjw2"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358668    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-host-run-netns\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358674    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b993b313-db8b-42f5-ad99-2d2ab5bb49e0-system-cni-dir\") pod \"multus-additional-cni-plugins-bsgc5\" (UID: \"b993b313-db8b-42f5-ad99-2d2ab5bb49e0\") " pod="openshift-multus/multus-additional-cni-plugins-bsgc5"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358682    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-etc-openvswitch\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358696    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-run\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358712    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-lib-modules\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358725    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-etc-openvswitch\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358736    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30-dbus\") pod \"global-pull-secret-syncer-4xjw2\" (UID: \"f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30\") " pod="kube-system/global-pull-secret-syncer-4xjw2"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358742    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d59f146b-82ee-4f26-854a-cba274a89600-device-dir\") pod \"aws-ebs-csi-driver-node-v6bks\" (UID: \"d59f146b-82ee-4f26-854a-cba274a89600\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6bks"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358745    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-run\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358743    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-host-run-netns\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358767    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a831ae97-8f46-4b85-bfdb-59042b7b8b0c-hosts-file\") pod \"node-resolver-9j547\" (UID: \"a831ae97-8f46-4b85-bfdb-59042b7b8b0c\") " pod="openshift-dns/node-resolver-9j547"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358773    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d59f146b-82ee-4f26-854a-cba274a89600-device-dir\") pod \"aws-ebs-csi-driver-node-v6bks\" (UID: \"d59f146b-82ee-4f26-854a-cba274a89600\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6bks"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358779    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-etc-sysconfig\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358790    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d59f146b-82ee-4f26-854a-cba274a89600-registration-dir\") pod \"aws-ebs-csi-driver-node-v6bks\" (UID: \"d59f146b-82ee-4f26-854a-cba274a89600\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6bks"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358809    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a831ae97-8f46-4b85-bfdb-59042b7b8b0c-hosts-file\") pod \"node-resolver-9j547\" (UID: \"a831ae97-8f46-4b85-bfdb-59042b7b8b0c\") " pod="openshift-dns/node-resolver-9j547"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358814    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-host-run-multus-certs\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358829    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d59f146b-82ee-4f26-854a-cba274a89600-registration-dir\") pod \"aws-ebs-csi-driver-node-v6bks\" (UID: \"d59f146b-82ee-4f26-854a-cba274a89600\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6bks"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358841    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-log-socket\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358816    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-lib-modules\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358860    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/778383aa-b3db-45a6-8bec-9891f7818e94-host-slash\") pod \"iptables-alerter-n9qsq\" (UID: \"778383aa-b3db-45a6-8bec-9891f7818e94\") " pod="openshift-network-operator/iptables-alerter-n9qsq"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358871    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-host-run-multus-certs\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358875    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d59f146b-82ee-4f26-854a-cba274a89600-etc-selinux\") pod \"aws-ebs-csi-driver-node-v6bks\" (UID: \"d59f146b-82ee-4f26-854a-cba274a89600\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6bks"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358906    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d59f146b-82ee-4f26-854a-cba274a89600-etc-selinux\") pod \"aws-ebs-csi-driver-node-v6bks\" (UID: \"d59f146b-82ee-4f26-854a-cba274a89600\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6bks"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358909    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/778383aa-b3db-45a6-8bec-9891f7818e94-host-slash\") pod \"iptables-alerter-n9qsq\" (UID: \"778383aa-b3db-45a6-8bec-9891f7818e94\") " pod="openshift-network-operator/iptables-alerter-n9qsq"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358885    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-log-socket\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358913    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-host-run-netns\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358934    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-cnibin\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358944    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-host-run-netns\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358951    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-multus-socket-dir-parent\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358983    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-node-log\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358995    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-cnibin\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.358998    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-multus-socket-dir-parent\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.359007    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-run-systemd\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.359034    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-node-log\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.359040    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-run-systemd\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.359036    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-host-cni-bin\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.359066    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-host-cni-bin\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.359074    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-etc-sysctl-d\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.359127    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d59f146b-82ee-4f26-854a-cba274a89600-socket-dir\") pod \"aws-ebs-csi-driver-node-v6bks\" (UID: \"d59f146b-82ee-4f26-854a-cba274a89600\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6bks"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.359144    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-multus-conf-dir\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.359158    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-host-run-ovn-kubernetes\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.359162    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-etc-sysctl-d\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.359172    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79212f04-d10c-43ee-9dd5-41b4fd1917b2-host\") pod \"node-ca-x8f2p\" (UID: \"79212f04-d10c-43ee-9dd5-41b4fd1917b2\") " pod="openshift-image-registry/node-ca-x8f2p"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.359185    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-multus-conf-dir\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.359190    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-sys\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.359209    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/55b54081-644c-464e-91e1-15c2d8f5be59-sys\") pod \"node-exporter-7w9wb\" (UID: \"55b54081-644c-464e-91e1-15c2d8f5be59\") " pod="openshift-monitoring/node-exporter-7w9wb"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.359222    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/55b54081-644c-464e-91e1-15c2d8f5be59-root\") pod \"node-exporter-7w9wb\" (UID: \"55b54081-644c-464e-91e1-15c2d8f5be59\") " pod="openshift-monitoring/node-exporter-7w9wb"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.359225    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d59f146b-82ee-4f26-854a-cba274a89600-socket-dir\") pod \"aws-ebs-csi-driver-node-v6bks\" (UID: \"d59f146b-82ee-4f26-854a-cba274a89600\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6bks"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.359233    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79212f04-d10c-43ee-9dd5-41b4fd1917b2-host\") pod \"node-ca-x8f2p\" (UID: \"79212f04-d10c-43ee-9dd5-41b4fd1917b2\") " pod="openshift-image-registry/node-ca-x8f2p"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.359249    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/55b54081-644c-464e-91e1-15c2d8f5be59-root\") pod \"node-exporter-7w9wb\" (UID: \"55b54081-644c-464e-91e1-15c2d8f5be59\") " pod="openshift-monitoring/node-exporter-7w9wb"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.359278    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-host-run-ovn-kubernetes\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.359281    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-sys\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.359248    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/55b54081-644c-464e-91e1-15c2d8f5be59-sys\") pod \"node-exporter-7w9wb\" (UID: \"55b54081-644c-464e-91e1-15c2d8f5be59\") " pod="openshift-monitoring/node-exporter-7w9wb"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.359307    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d59f146b-82ee-4f26-854a-cba274a89600-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v6bks\" (UID: \"d59f146b-82ee-4f26-854a-cba274a89600\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6bks"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.359336    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-host\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.359359    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d59f146b-82ee-4f26-854a-cba274a89600-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v6bks\" (UID: \"d59f146b-82ee-4f26-854a-cba274a89600\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6bks"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.359360    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-var-lib-openvswitch\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.359391    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-var-lib-openvswitch\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.359401    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-host\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.359394    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-etc-systemd\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.359417    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-etc-systemd\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.359429    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-host-run-k8s-cni-cncf-io\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.359458    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-etc-sysctl-conf\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.359470    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e-host-run-k8s-cni-cncf-io\") pod \"multus-jbbwp\" (UID: \"5a8c802b-6b86-41cd-ae81-bb3ed0b6ba4e\") " pod="openshift-multus/multus-jbbwp"
Nov 03 17:46:02.359461 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.359478    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-host-kubelet\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.361551 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.359532    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c29d200a-c917-46e5-9cf5-21f4d952e470-host-kubelet\") pod \"ovnkube-node-97p65\" (UID: \"c29d200a-c917-46e5-9cf5-21f4d952e470\") " pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:02.361551 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.359566    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ce15fea2-6b11-4b2b-8abb-7ec29840c78f-etc-sysctl-conf\") pod \"tuned-fjrg4\" (UID: \"ce15fea2-6b11-4b2b-8abb-7ec29840c78f\") " pod="openshift-cluster-node-tuning-operator/tuned-fjrg4"
Nov 03 17:46:02.513442 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:02.513400    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pf6z7"
Nov 03 17:46:04.023008 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:04.022971    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2t57w"
Nov 03 17:46:04.097734 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:04.097693    5315 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dv4zl_b91c00dc-c14f-4b3a-b268-1eb6dcf9ad4b/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:04.109187 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:04.109165    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-dwxs2"
Nov 03 17:46:04.112694 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:04.112672    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-dwxs2"
Nov 03 17:46:04.378663 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:04.378628    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:05.825663 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:05.825630    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-74479fcb5-w59vr"
Nov 03 17:46:07.991385 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:07.991356    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:07.991733 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:07.991593    5315 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 17:46:10.841865 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:10.841837    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-97p65"
Nov 03 17:46:11.813100 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:11.813070    5315 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-204.us-west-2.compute.internal" event="NodeReady"
Nov 03 17:46:24.575934 ip-10-0-128-204 kubenswrapper[5315]: E1103 17:46:24.575889    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 17:46:28.576053 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:28.576019    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dv4zl"
Nov 03 17:46:34.026366 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:46:34.026333    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2t57w"
Nov 03 17:46:54.575725 ip-10-0-128-204 kubenswrapper[5315]: E1103 17:46:54.575682    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 17:47:24.575741 ip-10-0-128-204 kubenswrapper[5315]: E1103 17:47:24.575701    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 17:47:54.576078 ip-10-0-128-204 kubenswrapper[5315]: E1103 17:47:54.576002    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 17:48:24.576457 ip-10-0-128-204 kubenswrapper[5315]: E1103 17:48:24.576421    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 17:48:52.721899 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:52.721859    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hckb"]
Nov 03 17:48:52.723883 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:52.723863    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hckb"
Nov 03 17:48:52.773672 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:52.773642    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 03 17:48:52.773809 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:52.773699    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 03 17:48:52.773809 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:52.773702    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-z9hfg\""
Nov 03 17:48:52.820220 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:52.820188    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hckb"]
Nov 03 17:48:52.846983 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:52.846955    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-k67zm"]
Nov 03 17:48:52.849139 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:52.849122    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k67zm"
Nov 03 17:48:52.867512 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:52.867488    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21a51980-64e9-4db8-9a16-ad412c14939e-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hckb\" (UID: \"21a51980-64e9-4db8-9a16-ad412c14939e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hckb"
Nov 03 17:48:52.867639 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:52.867539    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21a51980-64e9-4db8-9a16-ad412c14939e-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hckb\" (UID: \"21a51980-64e9-4db8-9a16-ad412c14939e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hckb"
Nov 03 17:48:52.867639 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:52.867575    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5f8t\" (UniqueName: \"kubernetes.io/projected/21a51980-64e9-4db8-9a16-ad412c14939e-kube-api-access-c5f8t\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hckb\" (UID: \"21a51980-64e9-4db8-9a16-ad412c14939e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hckb"
Nov 03 17:48:52.900678 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:52.900650    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 17:48:52.900826 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:52.900705    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 17:48:52.900826 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:52.900767    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 17:48:52.900927 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:52.900843    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 17:48:52.900993 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:52.900981    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 17:48:52.901033 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:52.901010    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 17:48:52.901066 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:52.901040    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-xxxzp\""
Nov 03 17:48:52.955395 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:52.955365    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-k67zm"]
Nov 03 17:48:52.968185 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:52.968160    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cc1495e5-5a42-4d6a-b2ce-c32090fdca5c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-k67zm\" (UID: \"cc1495e5-5a42-4d6a-b2ce-c32090fdca5c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k67zm"
Nov 03 17:48:52.968365 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:52.968200    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21a51980-64e9-4db8-9a16-ad412c14939e-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hckb\" (UID: \"21a51980-64e9-4db8-9a16-ad412c14939e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hckb"
Nov 03 17:48:52.968365 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:52.968224    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5f8t\" (UniqueName: \"kubernetes.io/projected/21a51980-64e9-4db8-9a16-ad412c14939e-kube-api-access-c5f8t\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hckb\" (UID: \"21a51980-64e9-4db8-9a16-ad412c14939e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hckb"
Nov 03 17:48:52.968365 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:52.968307    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cc1495e5-5a42-4d6a-b2ce-c32090fdca5c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-k67zm\" (UID: \"cc1495e5-5a42-4d6a-b2ce-c32090fdca5c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k67zm"
Nov 03 17:48:52.968365 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:52.968350    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc1495e5-5a42-4d6a-b2ce-c32090fdca5c-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-k67zm\" (UID: \"cc1495e5-5a42-4d6a-b2ce-c32090fdca5c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k67zm"
Nov 03 17:48:52.968588 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:52.968388    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21a51980-64e9-4db8-9a16-ad412c14939e-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hckb\" (UID: \"21a51980-64e9-4db8-9a16-ad412c14939e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hckb"
Nov 03 17:48:52.968588 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:52.968441    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw7dl\" (UniqueName: \"kubernetes.io/projected/cc1495e5-5a42-4d6a-b2ce-c32090fdca5c-kube-api-access-vw7dl\") pod \"istiod-openshift-gateway-7876bfcb74-k67zm\" (UID: \"cc1495e5-5a42-4d6a-b2ce-c32090fdca5c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k67zm"
Nov 03 17:48:52.968588 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:52.968471    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc1495e5-5a42-4d6a-b2ce-c32090fdca5c-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-k67zm\" (UID: \"cc1495e5-5a42-4d6a-b2ce-c32090fdca5c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k67zm"
Nov 03 17:48:52.968588 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:52.968493    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cc1495e5-5a42-4d6a-b2ce-c32090fdca5c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-k67zm\" (UID: \"cc1495e5-5a42-4d6a-b2ce-c32090fdca5c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k67zm"
Nov 03 17:48:52.968588 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:52.968519    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cc1495e5-5a42-4d6a-b2ce-c32090fdca5c-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-k67zm\" (UID: \"cc1495e5-5a42-4d6a-b2ce-c32090fdca5c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k67zm"
Nov 03 17:48:52.968878 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:52.968744    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21a51980-64e9-4db8-9a16-ad412c14939e-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hckb\" (UID: \"21a51980-64e9-4db8-9a16-ad412c14939e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hckb"
Nov 03 17:48:52.968878 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:52.968815    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21a51980-64e9-4db8-9a16-ad412c14939e-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hckb\" (UID: \"21a51980-64e9-4db8-9a16-ad412c14939e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hckb"
Nov 03 17:48:53.031654 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:53.031628    5315 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 17:48:53.033943 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:53.033924    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5f8t\" (UniqueName: \"kubernetes.io/projected/21a51980-64e9-4db8-9a16-ad412c14939e-kube-api-access-c5f8t\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hckb\" (UID: \"21a51980-64e9-4db8-9a16-ad412c14939e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hckb"
Nov 03 17:48:53.035553 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:53.035536    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hckb"
Nov 03 17:48:53.069703 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:53.069669    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cc1495e5-5a42-4d6a-b2ce-c32090fdca5c-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-k67zm\" (UID: \"cc1495e5-5a42-4d6a-b2ce-c32090fdca5c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k67zm"
Nov 03 17:48:53.069871 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:53.069712    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cc1495e5-5a42-4d6a-b2ce-c32090fdca5c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-k67zm\" (UID: \"cc1495e5-5a42-4d6a-b2ce-c32090fdca5c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k67zm"
Nov 03 17:48:53.069871 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:53.069846    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cc1495e5-5a42-4d6a-b2ce-c32090fdca5c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-k67zm\" (UID: \"cc1495e5-5a42-4d6a-b2ce-c32090fdca5c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k67zm"
Nov 03 17:48:53.069963 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:53.069897    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc1495e5-5a42-4d6a-b2ce-c32090fdca5c-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-k67zm\" (UID: \"cc1495e5-5a42-4d6a-b2ce-c32090fdca5c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k67zm"
Nov 03 17:48:53.069963 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:53.069942    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw7dl\" (UniqueName: \"kubernetes.io/projected/cc1495e5-5a42-4d6a-b2ce-c32090fdca5c-kube-api-access-vw7dl\") pod \"istiod-openshift-gateway-7876bfcb74-k67zm\" (UID: \"cc1495e5-5a42-4d6a-b2ce-c32090fdca5c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k67zm"
Nov 03 17:48:53.070057 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:53.069968    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc1495e5-5a42-4d6a-b2ce-c32090fdca5c-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-k67zm\" (UID: \"cc1495e5-5a42-4d6a-b2ce-c32090fdca5c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k67zm"
Nov 03 17:48:53.070057 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:53.069994    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cc1495e5-5a42-4d6a-b2ce-c32090fdca5c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-k67zm\" (UID: \"cc1495e5-5a42-4d6a-b2ce-c32090fdca5c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k67zm"
Nov 03 17:48:53.070804 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:53.070769    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cc1495e5-5a42-4d6a-b2ce-c32090fdca5c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-k67zm\" (UID: \"cc1495e5-5a42-4d6a-b2ce-c32090fdca5c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k67zm"
Nov 03 17:48:53.073103 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:53.073082    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cc1495e5-5a42-4d6a-b2ce-c32090fdca5c-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-k67zm\" (UID: \"cc1495e5-5a42-4d6a-b2ce-c32090fdca5c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k67zm"
Nov 03 17:48:53.073223 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:53.073173    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cc1495e5-5a42-4d6a-b2ce-c32090fdca5c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-k67zm\" (UID: \"cc1495e5-5a42-4d6a-b2ce-c32090fdca5c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k67zm"
Nov 03 17:48:53.073223 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:53.073200    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc1495e5-5a42-4d6a-b2ce-c32090fdca5c-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-k67zm\" (UID: \"cc1495e5-5a42-4d6a-b2ce-c32090fdca5c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k67zm"
Nov 03 17:48:53.073508 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:53.073489    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cc1495e5-5a42-4d6a-b2ce-c32090fdca5c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-k67zm\" (UID: \"cc1495e5-5a42-4d6a-b2ce-c32090fdca5c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k67zm"
Nov 03 17:48:53.132201 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:53.132169    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc1495e5-5a42-4d6a-b2ce-c32090fdca5c-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-k67zm\" (UID: \"cc1495e5-5a42-4d6a-b2ce-c32090fdca5c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k67zm"
Nov 03 17:48:53.133131 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:53.133108    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw7dl\" (UniqueName: \"kubernetes.io/projected/cc1495e5-5a42-4d6a-b2ce-c32090fdca5c-kube-api-access-vw7dl\") pod \"istiod-openshift-gateway-7876bfcb74-k67zm\" (UID: \"cc1495e5-5a42-4d6a-b2ce-c32090fdca5c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k67zm"
Nov 03 17:48:53.159218 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:53.159191    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k67zm"
Nov 03 17:48:53.417074 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:53.417006    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hckb"]
Nov 03 17:48:53.424815 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:48:53.424786    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21a51980_64e9_4db8_9a16_ad412c14939e.slice/crio-3f81213dc1c13255ca93067b45d2978dc14d265129994c5a6ac3cb7e291b3693 WatchSource:0}: Error finding container 3f81213dc1c13255ca93067b45d2978dc14d265129994c5a6ac3cb7e291b3693: Status 404 returned error can't find the container with id 3f81213dc1c13255ca93067b45d2978dc14d265129994c5a6ac3cb7e291b3693
Nov 03 17:48:53.426833 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:53.426816    5315 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 17:48:53.535478 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:53.535442    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-k67zm"]
Nov 03 17:48:53.540508 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:48:53.540483    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc1495e5_5a42_4d6a_b2ce_c32090fdca5c.slice/crio-119a66783219c465865b5c570bd68762bbd662e973321177a16dbad1f703b948 WatchSource:0}: Error finding container 119a66783219c465865b5c570bd68762bbd662e973321177a16dbad1f703b948: Status 404 returned error can't find the container with id 119a66783219c465865b5c570bd68762bbd662e973321177a16dbad1f703b948
Nov 03 17:48:53.689092 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:53.688995    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k67zm" event={"ID":"cc1495e5-5a42-4d6a-b2ce-c32090fdca5c","Type":"ContainerStarted","Data":"119a66783219c465865b5c570bd68762bbd662e973321177a16dbad1f703b948"}
Nov 03 17:48:53.690142 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:53.690117    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hckb" event={"ID":"21a51980-64e9-4db8-9a16-ad412c14939e","Type":"ContainerStarted","Data":"3f81213dc1c13255ca93067b45d2978dc14d265129994c5a6ac3cb7e291b3693"}
Nov 03 17:48:54.579277 ip-10-0-128-204 kubenswrapper[5315]: E1103 17:48:54.579237    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 17:48:58.385828 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:58.385793    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 17:48:58.386088 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:58.385871    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 17:48:58.708764 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:58.708668    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k67zm" event={"ID":"cc1495e5-5a42-4d6a-b2ce-c32090fdca5c","Type":"ContainerStarted","Data":"771a43056625f47d455411557369d862880a7e8872b19fb1a017ded92ad870b8"}
Nov 03 17:48:58.708932 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:58.708788    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k67zm"
Nov 03 17:48:58.868354 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:58.868302    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k67zm" podStartSLOduration=2.025934587 podStartE2EDuration="6.86828855s" totalImagesPullingTime="4.842353963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:48:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:48:58.867923771 +0000 UTC m=+177.884121841" watchObservedRunningTime="2026-11-03 17:48:58.86828855 +0000 UTC m=+177.884486596"
Nov 03 17:48:59.714468 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:59.714423    5315 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-k67zm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 17:48:59.714961 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:48:59.714488    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k67zm" podUID="cc1495e5-5a42-4d6a-b2ce-c32090fdca5c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 17:49:02.713863 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:02.713823    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k67zm"
Nov 03 17:49:05.733930 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:05.733835    5315 generic.go:356] "Generic (PLEG): container finished" podID="21a51980-64e9-4db8-9a16-ad412c14939e" containerID="de49a5926ef5a9693803215abd9cd7bd35136796691137a10a254c23f0144442" exitCode=0
Nov 03 17:49:05.733930 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:05.733924    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hckb" event={"ID":"21a51980-64e9-4db8-9a16-ad412c14939e","Type":"ContainerDied","Data":"de49a5926ef5a9693803215abd9cd7bd35136796691137a10a254c23f0144442"}
Nov 03 17:49:10.755673 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:10.755639    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hckb" event={"ID":"21a51980-64e9-4db8-9a16-ad412c14939e","Type":"ContainerStarted","Data":"1cde97dd4b8d8ce9e886537830a4efab10ad5bc4cbe531255a9fb5d1e39076b0"}
Nov 03 17:49:11.760036 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:11.760001    5315 generic.go:356] "Generic (PLEG): container finished" podID="21a51980-64e9-4db8-9a16-ad412c14939e" containerID="1cde97dd4b8d8ce9e886537830a4efab10ad5bc4cbe531255a9fb5d1e39076b0" exitCode=0
Nov 03 17:49:11.760421 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:11.760090    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hckb" event={"ID":"21a51980-64e9-4db8-9a16-ad412c14939e","Type":"ContainerDied","Data":"1cde97dd4b8d8ce9e886537830a4efab10ad5bc4cbe531255a9fb5d1e39076b0"}
Nov 03 17:49:19.789493 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:19.789455    5315 generic.go:356] "Generic (PLEG): container finished" podID="21a51980-64e9-4db8-9a16-ad412c14939e" containerID="d7032425d97be5d20189d2537ae36073765b554a19838ca68602adb4b0c2721e" exitCode=0
Nov 03 17:49:19.789942 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:19.789545    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hckb" event={"ID":"21a51980-64e9-4db8-9a16-ad412c14939e","Type":"ContainerDied","Data":"d7032425d97be5d20189d2537ae36073765b554a19838ca68602adb4b0c2721e"}
Nov 03 17:49:20.952058 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:20.952029    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hckb"
Nov 03 17:49:20.990735 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:20.990706    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21a51980-64e9-4db8-9a16-ad412c14939e-util\" (UniqueName: \"kubernetes.io/empty-dir/21a51980-64e9-4db8-9a16-ad412c14939e-util\") pod \"21a51980-64e9-4db8-9a16-ad412c14939e\" (UID: \"21a51980-64e9-4db8-9a16-ad412c14939e\") "
Nov 03 17:49:20.990901 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:20.990744    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21a51980-64e9-4db8-9a16-ad412c14939e-kube-api-access-c5f8t\" (UniqueName: \"kubernetes.io/projected/21a51980-64e9-4db8-9a16-ad412c14939e-kube-api-access-c5f8t\") pod \"21a51980-64e9-4db8-9a16-ad412c14939e\" (UID: \"21a51980-64e9-4db8-9a16-ad412c14939e\") "
Nov 03 17:49:20.990901 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:20.990799    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21a51980-64e9-4db8-9a16-ad412c14939e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/21a51980-64e9-4db8-9a16-ad412c14939e-bundle\") pod \"21a51980-64e9-4db8-9a16-ad412c14939e\" (UID: \"21a51980-64e9-4db8-9a16-ad412c14939e\") "
Nov 03 17:49:20.991376 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:20.991356    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21a51980-64e9-4db8-9a16-ad412c14939e-bundle" pod "21a51980-64e9-4db8-9a16-ad412c14939e" (UID: "21a51980-64e9-4db8-9a16-ad412c14939e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:49:20.992903 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:20.992885    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21a51980-64e9-4db8-9a16-ad412c14939e-kube-api-access-c5f8t" pod "21a51980-64e9-4db8-9a16-ad412c14939e" (UID: "21a51980-64e9-4db8-9a16-ad412c14939e"). InnerVolumeSpecName "kube-api-access-c5f8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:49:20.995046 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:20.995021    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21a51980-64e9-4db8-9a16-ad412c14939e-util" pod "21a51980-64e9-4db8-9a16-ad412c14939e" (UID: "21a51980-64e9-4db8-9a16-ad412c14939e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:49:21.091803 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:21.091714    5315 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21a51980-64e9-4db8-9a16-ad412c14939e-bundle\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:21.091803 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:21.091743    5315 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21a51980-64e9-4db8-9a16-ad412c14939e-util\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:21.091803 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:21.091766    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5f8t\" (UniqueName: \"kubernetes.io/projected/21a51980-64e9-4db8-9a16-ad412c14939e-kube-api-access-c5f8t\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:21.799023 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:21.798989    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hckb" event={"ID":"21a51980-64e9-4db8-9a16-ad412c14939e","Type":"ContainerDied","Data":"3f81213dc1c13255ca93067b45d2978dc14d265129994c5a6ac3cb7e291b3693"}
Nov 03 17:49:21.799023 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:21.799023    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8hckb"
Nov 03 17:49:21.799222 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:21.799037    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f81213dc1c13255ca93067b45d2978dc14d265129994c5a6ac3cb7e291b3693"
Nov 03 17:49:24.576248 ip-10-0-128-204 kubenswrapper[5315]: E1103 17:49:24.576201    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 17:49:40.278432 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:40.278390    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-b8vdv"]
Nov 03 17:49:40.278962 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:40.278783    5315 cpu_manager.go:414] "containerMap: removing container" podUID="21a51980-64e9-4db8-9a16-ad412c14939e" containerName="util"
Nov 03 17:49:40.278962 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:40.278802    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21a51980-64e9-4db8-9a16-ad412c14939e" containerName="util"
Nov 03 17:49:40.278962 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:40.278849    5315 cpu_manager.go:414] "containerMap: removing container" podUID="21a51980-64e9-4db8-9a16-ad412c14939e" containerName="pull"
Nov 03 17:49:40.278962 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:40.278857    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21a51980-64e9-4db8-9a16-ad412c14939e" containerName="pull"
Nov 03 17:49:40.278962 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:40.278868    5315 cpu_manager.go:414] "containerMap: removing container" podUID="21a51980-64e9-4db8-9a16-ad412c14939e" containerName="extract"
Nov 03 17:49:40.278962 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:40.278875    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21a51980-64e9-4db8-9a16-ad412c14939e" containerName="extract"
Nov 03 17:49:40.278962 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:40.278937    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="21a51980-64e9-4db8-9a16-ad412c14939e" containerName="extract"
Nov 03 17:49:40.281070 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:40.281050    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-b8vdv"
Nov 03 17:49:40.387893 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:40.387857    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 03 17:49:40.388070 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:40.387898    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 03 17:49:40.388070 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:40.387923    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 03 17:49:40.388070 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:40.387973    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-jdbrd\""
Nov 03 17:49:40.388070 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:40.388036    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 03 17:49:40.388070 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:40.388056    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 03 17:49:40.389002 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:40.388986    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-b8vdv"]
Nov 03 17:49:40.427699 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:40.427669    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/795139d5-8598-4339-aaf1-32608d92c270-manifests\") pod \"opendatahub-operator-controller-manager-6dd86677fd-b8vdv\" (UID: \"795139d5-8598-4339-aaf1-32608d92c270\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-b8vdv"
Nov 03 17:49:40.427699 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:40.427698    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/795139d5-8598-4339-aaf1-32608d92c270-tmp\") pod \"opendatahub-operator-controller-manager-6dd86677fd-b8vdv\" (UID: \"795139d5-8598-4339-aaf1-32608d92c270\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-b8vdv"
Nov 03 17:49:40.427867 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:40.427722    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/795139d5-8598-4339-aaf1-32608d92c270-metrics-cert\") pod \"opendatahub-operator-controller-manager-6dd86677fd-b8vdv\" (UID: \"795139d5-8598-4339-aaf1-32608d92c270\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-b8vdv"
Nov 03 17:49:40.427867 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:40.427738    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/795139d5-8598-4339-aaf1-32608d92c270-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dd86677fd-b8vdv\" (UID: \"795139d5-8598-4339-aaf1-32608d92c270\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-b8vdv"
Nov 03 17:49:40.427867 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:40.427848    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/795139d5-8598-4339-aaf1-32608d92c270-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dd86677fd-b8vdv\" (UID: \"795139d5-8598-4339-aaf1-32608d92c270\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-b8vdv"
Nov 03 17:49:40.427960 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:40.427869    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7nrj\" (UniqueName: \"kubernetes.io/projected/795139d5-8598-4339-aaf1-32608d92c270-kube-api-access-d7nrj\") pod \"opendatahub-operator-controller-manager-6dd86677fd-b8vdv\" (UID: \"795139d5-8598-4339-aaf1-32608d92c270\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-b8vdv"
Nov 03 17:49:40.528342 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:40.528305    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/795139d5-8598-4339-aaf1-32608d92c270-manifests\") pod \"opendatahub-operator-controller-manager-6dd86677fd-b8vdv\" (UID: \"795139d5-8598-4339-aaf1-32608d92c270\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-b8vdv"
Nov 03 17:49:40.528342 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:40.528342    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/795139d5-8598-4339-aaf1-32608d92c270-tmp\") pod \"opendatahub-operator-controller-manager-6dd86677fd-b8vdv\" (UID: \"795139d5-8598-4339-aaf1-32608d92c270\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-b8vdv"
Nov 03 17:49:40.528837 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:40.528371    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/795139d5-8598-4339-aaf1-32608d92c270-metrics-cert\") pod \"opendatahub-operator-controller-manager-6dd86677fd-b8vdv\" (UID: \"795139d5-8598-4339-aaf1-32608d92c270\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-b8vdv"
Nov 03 17:49:40.528837 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:40.528492    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/795139d5-8598-4339-aaf1-32608d92c270-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dd86677fd-b8vdv\" (UID: \"795139d5-8598-4339-aaf1-32608d92c270\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-b8vdv"
Nov 03 17:49:40.528837 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:40.528536    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/795139d5-8598-4339-aaf1-32608d92c270-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dd86677fd-b8vdv\" (UID: \"795139d5-8598-4339-aaf1-32608d92c270\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-b8vdv"
Nov 03 17:49:40.528837 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:40.528571    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7nrj\" (UniqueName: \"kubernetes.io/projected/795139d5-8598-4339-aaf1-32608d92c270-kube-api-access-d7nrj\") pod \"opendatahub-operator-controller-manager-6dd86677fd-b8vdv\" (UID: \"795139d5-8598-4339-aaf1-32608d92c270\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-b8vdv"
Nov 03 17:49:40.529074 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:40.528837    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/795139d5-8598-4339-aaf1-32608d92c270-manifests\") pod \"opendatahub-operator-controller-manager-6dd86677fd-b8vdv\" (UID: \"795139d5-8598-4339-aaf1-32608d92c270\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-b8vdv"
Nov 03 17:49:40.530706 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:40.530683    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/795139d5-8598-4339-aaf1-32608d92c270-tmp\") pod \"opendatahub-operator-controller-manager-6dd86677fd-b8vdv\" (UID: \"795139d5-8598-4339-aaf1-32608d92c270\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-b8vdv"
Nov 03 17:49:40.530911 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:40.530895    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/795139d5-8598-4339-aaf1-32608d92c270-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dd86677fd-b8vdv\" (UID: \"795139d5-8598-4339-aaf1-32608d92c270\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-b8vdv"
Nov 03 17:49:40.531087 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:40.531066    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/795139d5-8598-4339-aaf1-32608d92c270-metrics-cert\") pod \"opendatahub-operator-controller-manager-6dd86677fd-b8vdv\" (UID: \"795139d5-8598-4339-aaf1-32608d92c270\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-b8vdv"
Nov 03 17:49:40.531125 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:40.531067    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/795139d5-8598-4339-aaf1-32608d92c270-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dd86677fd-b8vdv\" (UID: \"795139d5-8598-4339-aaf1-32608d92c270\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-b8vdv"
Nov 03 17:49:40.611879 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:40.611852    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7nrj\" (UniqueName: \"kubernetes.io/projected/795139d5-8598-4339-aaf1-32608d92c270-kube-api-access-d7nrj\") pod \"opendatahub-operator-controller-manager-6dd86677fd-b8vdv\" (UID: \"795139d5-8598-4339-aaf1-32608d92c270\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-b8vdv"
Nov 03 17:49:40.893450 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:40.893367    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-b8vdv"
Nov 03 17:49:41.476862 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:41.476781    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-b8vdv"]
Nov 03 17:49:41.481160 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:49:41.481118    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod795139d5_8598_4339_aaf1_32608d92c270.slice/crio-693ac38a13af8cd26671ad3450e1fd07674980421d617c6d47b03aaab87f04ad WatchSource:0}: Error finding container 693ac38a13af8cd26671ad3450e1fd07674980421d617c6d47b03aaab87f04ad: Status 404 returned error can't find the container with id 693ac38a13af8cd26671ad3450e1fd07674980421d617c6d47b03aaab87f04ad
Nov 03 17:49:41.863438 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:41.863403    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-b8vdv" event={"ID":"795139d5-8598-4339-aaf1-32608d92c270","Type":"ContainerStarted","Data":"693ac38a13af8cd26671ad3450e1fd07674980421d617c6d47b03aaab87f04ad"}
Nov 03 17:49:44.876037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:44.876000    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-b8vdv" event={"ID":"795139d5-8598-4339-aaf1-32608d92c270","Type":"ContainerStarted","Data":"5d947cddded4a8af631265c0b53724feaec661678c302e72dfcc55cca085becd"}
Nov 03 17:49:45.880786 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:45.880748    5315 generic.go:356] "Generic (PLEG): container finished" podID="795139d5-8598-4339-aaf1-32608d92c270" containerID="5d947cddded4a8af631265c0b53724feaec661678c302e72dfcc55cca085becd" exitCode=0
Nov 03 17:49:45.881195 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:45.880792    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-b8vdv" event={"ID":"795139d5-8598-4339-aaf1-32608d92c270","Type":"ContainerDied","Data":"5d947cddded4a8af631265c0b53724feaec661678c302e72dfcc55cca085becd"}
Nov 03 17:49:46.886273 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:46.886232    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-b8vdv" event={"ID":"795139d5-8598-4339-aaf1-32608d92c270","Type":"ContainerStarted","Data":"1f67c1cba1132a0645f7048b32b8cdd21730347213e435d439cb02f751a59213"}
Nov 03 17:49:46.886681 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:46.886340    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-b8vdv"
Nov 03 17:49:47.024546 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:47.024440    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-b8vdv" podStartSLOduration=2.492216798 podStartE2EDuration="7.02442458s" totalImagesPullingTime="4.059244682s" totalInitContainerRuntime="472.9631ms" isStatefulPod=false podCreationTimestamp="2026-11-03 17:49:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:49:47.020292012 +0000 UTC m=+226.036490082" watchObservedRunningTime="2026-11-03 17:49:47.02442458 +0000 UTC m=+226.040622716"
Nov 03 17:49:54.580319 ip-10-0-128-204 kubenswrapper[5315]: E1103 17:49:54.576581    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 17:49:57.892200 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:49:57.892168    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-b8vdv"
Nov 03 17:50:19.150318 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:19.150271    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-54df9ccd9f-vjs8r"]
Nov 03 17:50:19.157113 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:19.157089    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54df9ccd9f-vjs8r"
Nov 03 17:50:19.224913 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:19.224858    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2srd7\" (UniqueName: \"kubernetes.io/projected/1341e0b3-9782-4dae-9d05-7b77461f63ab-kube-api-access-2srd7\") pod \"kube-auth-proxy-54df9ccd9f-vjs8r\" (UID: \"1341e0b3-9782-4dae-9d05-7b77461f63ab\") " pod="openshift-ingress/kube-auth-proxy-54df9ccd9f-vjs8r"
Nov 03 17:50:19.225097 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:19.224987    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1341e0b3-9782-4dae-9d05-7b77461f63ab-tmp\") pod \"kube-auth-proxy-54df9ccd9f-vjs8r\" (UID: \"1341e0b3-9782-4dae-9d05-7b77461f63ab\") " pod="openshift-ingress/kube-auth-proxy-54df9ccd9f-vjs8r"
Nov 03 17:50:19.225178 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:19.225131    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1341e0b3-9782-4dae-9d05-7b77461f63ab-tls-certs\") pod \"kube-auth-proxy-54df9ccd9f-vjs8r\" (UID: \"1341e0b3-9782-4dae-9d05-7b77461f63ab\") " pod="openshift-ingress/kube-auth-proxy-54df9ccd9f-vjs8r"
Nov 03 17:50:19.254626 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:19.254589    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 03 17:50:19.254626 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:19.254605    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 03 17:50:19.254828 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:19.254704    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-nrzqr\""
Nov 03 17:50:19.255520 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:19.255499    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54df9ccd9f-vjs8r"]
Nov 03 17:50:19.326273 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:19.326243    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1341e0b3-9782-4dae-9d05-7b77461f63ab-tls-certs\") pod \"kube-auth-proxy-54df9ccd9f-vjs8r\" (UID: \"1341e0b3-9782-4dae-9d05-7b77461f63ab\") " pod="openshift-ingress/kube-auth-proxy-54df9ccd9f-vjs8r"
Nov 03 17:50:19.326463 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:19.326300    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2srd7\" (UniqueName: \"kubernetes.io/projected/1341e0b3-9782-4dae-9d05-7b77461f63ab-kube-api-access-2srd7\") pod \"kube-auth-proxy-54df9ccd9f-vjs8r\" (UID: \"1341e0b3-9782-4dae-9d05-7b77461f63ab\") " pod="openshift-ingress/kube-auth-proxy-54df9ccd9f-vjs8r"
Nov 03 17:50:19.326463 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:19.326348    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1341e0b3-9782-4dae-9d05-7b77461f63ab-tmp\") pod \"kube-auth-proxy-54df9ccd9f-vjs8r\" (UID: \"1341e0b3-9782-4dae-9d05-7b77461f63ab\") " pod="openshift-ingress/kube-auth-proxy-54df9ccd9f-vjs8r"
Nov 03 17:50:19.328571 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:19.328546    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1341e0b3-9782-4dae-9d05-7b77461f63ab-tmp\") pod \"kube-auth-proxy-54df9ccd9f-vjs8r\" (UID: \"1341e0b3-9782-4dae-9d05-7b77461f63ab\") " pod="openshift-ingress/kube-auth-proxy-54df9ccd9f-vjs8r"
Nov 03 17:50:19.328849 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:19.328827    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1341e0b3-9782-4dae-9d05-7b77461f63ab-tls-certs\") pod \"kube-auth-proxy-54df9ccd9f-vjs8r\" (UID: \"1341e0b3-9782-4dae-9d05-7b77461f63ab\") " pod="openshift-ingress/kube-auth-proxy-54df9ccd9f-vjs8r"
Nov 03 17:50:19.383967 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:19.383938    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2srd7\" (UniqueName: \"kubernetes.io/projected/1341e0b3-9782-4dae-9d05-7b77461f63ab-kube-api-access-2srd7\") pod \"kube-auth-proxy-54df9ccd9f-vjs8r\" (UID: \"1341e0b3-9782-4dae-9d05-7b77461f63ab\") " pod="openshift-ingress/kube-auth-proxy-54df9ccd9f-vjs8r"
Nov 03 17:50:19.471105 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:19.471029    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54df9ccd9f-vjs8r"
Nov 03 17:50:19.852349 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:19.852313    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54df9ccd9f-vjs8r"]
Nov 03 17:50:19.863005 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:50:19.862977    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1341e0b3_9782_4dae_9d05_7b77461f63ab.slice/crio-5c4f84ce4d658e463df92472f065d7272cfc26c35bca89e789ab41c4d43ece01 WatchSource:0}: Error finding container 5c4f84ce4d658e463df92472f065d7272cfc26c35bca89e789ab41c4d43ece01: Status 404 returned error can't find the container with id 5c4f84ce4d658e463df92472f065d7272cfc26c35bca89e789ab41c4d43ece01
Nov 03 17:50:19.994198 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:19.994160    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54df9ccd9f-vjs8r" event={"ID":"1341e0b3-9782-4dae-9d05-7b77461f63ab","Type":"ContainerStarted","Data":"5c4f84ce4d658e463df92472f065d7272cfc26c35bca89e789ab41c4d43ece01"}
Nov 03 17:50:22.237343 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:22.237302    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-8gsxp"]
Nov 03 17:50:22.242857 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:22.242828    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-8gsxp"
Nov 03 17:50:22.249991 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:22.249959    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9d8n\" (UniqueName: \"kubernetes.io/projected/8c11a31f-ff69-4b63-b672-3d32e92ed503-kube-api-access-f9d8n\") pod \"authorino-57f7f8659f-8gsxp\" (UID: \"8c11a31f-ff69-4b63-b672-3d32e92ed503\") " pod="kuadrant-system/authorino-57f7f8659f-8gsxp"
Nov 03 17:50:22.291562 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:22.291529    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-4hst9\""
Nov 03 17:50:22.291740 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:22.291565    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 03 17:50:22.337546 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:22.337517    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 03 17:50:22.338238 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:22.338214    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-8gsxp"]
Nov 03 17:50:22.351259 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:22.351215    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9d8n\" (UniqueName: \"kubernetes.io/projected/8c11a31f-ff69-4b63-b672-3d32e92ed503-kube-api-access-f9d8n\") pod \"authorino-57f7f8659f-8gsxp\" (UID: \"8c11a31f-ff69-4b63-b672-3d32e92ed503\") " pod="kuadrant-system/authorino-57f7f8659f-8gsxp"
Nov 03 17:50:22.408902 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:22.408870    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9d8n\" (UniqueName: \"kubernetes.io/projected/8c11a31f-ff69-4b63-b672-3d32e92ed503-kube-api-access-f9d8n\") pod \"authorino-57f7f8659f-8gsxp\" (UID: \"8c11a31f-ff69-4b63-b672-3d32e92ed503\") " pod="kuadrant-system/authorino-57f7f8659f-8gsxp"
Nov 03 17:50:22.574237 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:22.574201    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-8gsxp"
Nov 03 17:50:22.744786 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:22.744750    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-69b95cc89f-9vfx7"]
Nov 03 17:50:22.750448 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:22.750417    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-69b95cc89f-9vfx7"
Nov 03 17:50:22.818579 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:22.818547    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-69b95cc89f-9vfx7"]
Nov 03 17:50:22.848056 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:22.847983    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 03 17:50:22.856159 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:22.856131    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f94dc64-1830-4201-8a04-989ff3339195-metrics-certs\") pod \"jobset-controller-manager-69b95cc89f-9vfx7\" (UID: \"7f94dc64-1830-4201-8a04-989ff3339195\") " pod="openshift-jobset-operator/jobset-controller-manager-69b95cc89f-9vfx7"
Nov 03 17:50:22.856296 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:22.856167    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvgcx\" (UniqueName: \"kubernetes.io/projected/7f94dc64-1830-4201-8a04-989ff3339195-kube-api-access-dvgcx\") pod \"jobset-controller-manager-69b95cc89f-9vfx7\" (UID: \"7f94dc64-1830-4201-8a04-989ff3339195\") " pod="openshift-jobset-operator/jobset-controller-manager-69b95cc89f-9vfx7"
Nov 03 17:50:22.856296 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:22.856188    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7f94dc64-1830-4201-8a04-989ff3339195-cert\") pod \"jobset-controller-manager-69b95cc89f-9vfx7\" (UID: \"7f94dc64-1830-4201-8a04-989ff3339195\") " pod="openshift-jobset-operator/jobset-controller-manager-69b95cc89f-9vfx7"
Nov 03 17:50:22.856369 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:22.856328    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7f94dc64-1830-4201-8a04-989ff3339195-manager-config\") pod \"jobset-controller-manager-69b95cc89f-9vfx7\" (UID: \"7f94dc64-1830-4201-8a04-989ff3339195\") " pod="openshift-jobset-operator/jobset-controller-manager-69b95cc89f-9vfx7"
Nov 03 17:50:22.863345 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:22.863330    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Nov 03 17:50:22.863422 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:22.863384    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 03 17:50:22.863472 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:22.863460    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Nov 03 17:50:22.863510 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:22.863464    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Nov 03 17:50:22.863546 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:22.863524    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-hlk4m\""
Nov 03 17:50:22.957702 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:22.957659    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f94dc64-1830-4201-8a04-989ff3339195-metrics-certs\") pod \"jobset-controller-manager-69b95cc89f-9vfx7\" (UID: \"7f94dc64-1830-4201-8a04-989ff3339195\") " pod="openshift-jobset-operator/jobset-controller-manager-69b95cc89f-9vfx7"
Nov 03 17:50:22.957893 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:22.957719    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvgcx\" (UniqueName: \"kubernetes.io/projected/7f94dc64-1830-4201-8a04-989ff3339195-kube-api-access-dvgcx\") pod \"jobset-controller-manager-69b95cc89f-9vfx7\" (UID: \"7f94dc64-1830-4201-8a04-989ff3339195\") " pod="openshift-jobset-operator/jobset-controller-manager-69b95cc89f-9vfx7"
Nov 03 17:50:22.957893 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:22.957753    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7f94dc64-1830-4201-8a04-989ff3339195-cert\") pod \"jobset-controller-manager-69b95cc89f-9vfx7\" (UID: \"7f94dc64-1830-4201-8a04-989ff3339195\") " pod="openshift-jobset-operator/jobset-controller-manager-69b95cc89f-9vfx7"
Nov 03 17:50:22.957893 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:22.957787    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7f94dc64-1830-4201-8a04-989ff3339195-manager-config\") pod \"jobset-controller-manager-69b95cc89f-9vfx7\" (UID: \"7f94dc64-1830-4201-8a04-989ff3339195\") " pod="openshift-jobset-operator/jobset-controller-manager-69b95cc89f-9vfx7"
Nov 03 17:50:22.958689 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:22.958624    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7f94dc64-1830-4201-8a04-989ff3339195-manager-config\") pod \"jobset-controller-manager-69b95cc89f-9vfx7\" (UID: \"7f94dc64-1830-4201-8a04-989ff3339195\") " pod="openshift-jobset-operator/jobset-controller-manager-69b95cc89f-9vfx7"
Nov 03 17:50:22.960695 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:22.960672    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7f94dc64-1830-4201-8a04-989ff3339195-cert\") pod \"jobset-controller-manager-69b95cc89f-9vfx7\" (UID: \"7f94dc64-1830-4201-8a04-989ff3339195\") " pod="openshift-jobset-operator/jobset-controller-manager-69b95cc89f-9vfx7"
Nov 03 17:50:22.960831 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:22.960810    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f94dc64-1830-4201-8a04-989ff3339195-metrics-certs\") pod \"jobset-controller-manager-69b95cc89f-9vfx7\" (UID: \"7f94dc64-1830-4201-8a04-989ff3339195\") " pod="openshift-jobset-operator/jobset-controller-manager-69b95cc89f-9vfx7"
Nov 03 17:50:22.991275 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:22.991239    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-8gsxp"]
Nov 03 17:50:22.999594 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:50:22.999561    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c11a31f_ff69_4b63_b672_3d32e92ed503.slice/crio-f62a477627aad4925fe171a78e6f606036eed1fe579436f7d7a5d5c1eef9af32 WatchSource:0}: Error finding container f62a477627aad4925fe171a78e6f606036eed1fe579436f7d7a5d5c1eef9af32: Status 404 returned error can't find the container with id f62a477627aad4925fe171a78e6f606036eed1fe579436f7d7a5d5c1eef9af32
Nov 03 17:50:23.005198 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:23.005169    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-8gsxp" event={"ID":"8c11a31f-ff69-4b63-b672-3d32e92ed503","Type":"ContainerStarted","Data":"f62a477627aad4925fe171a78e6f606036eed1fe579436f7d7a5d5c1eef9af32"}
Nov 03 17:50:23.017594 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:23.017570    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvgcx\" (UniqueName: \"kubernetes.io/projected/7f94dc64-1830-4201-8a04-989ff3339195-kube-api-access-dvgcx\") pod \"jobset-controller-manager-69b95cc89f-9vfx7\" (UID: \"7f94dc64-1830-4201-8a04-989ff3339195\") " pod="openshift-jobset-operator/jobset-controller-manager-69b95cc89f-9vfx7"
Nov 03 17:50:23.066915 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:23.066882    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-69b95cc89f-9vfx7"
Nov 03 17:50:23.462137 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:23.462058    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-69b95cc89f-9vfx7"]
Nov 03 17:50:23.469941 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:50:23.469912    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f94dc64_1830_4201_8a04_989ff3339195.slice/crio-89a7ef777d39a7292000ae61c73d233bb9ca1f2085690dc59d9a345cdc447ef6 WatchSource:0}: Error finding container 89a7ef777d39a7292000ae61c73d233bb9ca1f2085690dc59d9a345cdc447ef6: Status 404 returned error can't find the container with id 89a7ef777d39a7292000ae61c73d233bb9ca1f2085690dc59d9a345cdc447ef6
Nov 03 17:50:24.009483 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:24.009443    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54df9ccd9f-vjs8r" event={"ID":"1341e0b3-9782-4dae-9d05-7b77461f63ab","Type":"ContainerStarted","Data":"73a9292a0b551adcbc627867eb99bbe9962e355a149167f11a21eaed294230d1"}
Nov 03 17:50:24.010602 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:24.010577    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-69b95cc89f-9vfx7" event={"ID":"7f94dc64-1830-4201-8a04-989ff3339195","Type":"ContainerStarted","Data":"89a7ef777d39a7292000ae61c73d233bb9ca1f2085690dc59d9a345cdc447ef6"}
Nov 03 17:50:24.122100 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:24.122052    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-54df9ccd9f-vjs8r" podStartSLOduration=1.704285181 podStartE2EDuration="5.122038564s" totalImagesPullingTime="3.417753383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:50:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:50:24.121401774 +0000 UTC m=+263.137599843" watchObservedRunningTime="2026-11-03 17:50:24.122038564 +0000 UTC m=+263.138236633"
Nov 03 17:50:24.575830 ip-10-0-128-204 kubenswrapper[5315]: E1103 17:50:24.575790    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 17:50:29.030640 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:29.030586    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-69b95cc89f-9vfx7" event={"ID":"7f94dc64-1830-4201-8a04-989ff3339195","Type":"ContainerStarted","Data":"17420816a8b2662ecacf0b4733a7c71d8c33650c5d5e70dd7c3aa7cd138d678a"}
Nov 03 17:50:29.031040 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:29.030684    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-69b95cc89f-9vfx7"
Nov 03 17:50:29.142871 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:29.142817    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-69b95cc89f-9vfx7" podStartSLOduration=2.594627141 podStartE2EDuration="7.142803185s" totalImagesPullingTime="4.548176044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:50:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:50:29.142069436 +0000 UTC m=+268.158267505" watchObservedRunningTime="2026-11-03 17:50:29.142803185 +0000 UTC m=+268.159001246"
Nov 03 17:50:29.637651 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:29.637600    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-8gsxp"]
Nov 03 17:50:40.039336 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:40.039299    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-69b95cc89f-9vfx7"
Nov 03 17:50:53.111687 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:53.111648    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-8gsxp" event={"ID":"8c11a31f-ff69-4b63-b672-3d32e92ed503","Type":"ContainerStarted","Data":"7a728334b34e45444d847c455c65c80bc81e0970b4156f5fd5599bd806112a0e"}
Nov 03 17:50:53.112208 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:53.111765    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-8gsxp" podUID="8c11a31f-ff69-4b63-b672-3d32e92ed503" containerName="authorino" containerID="cri-o://7a728334b34e45444d847c455c65c80bc81e0970b4156f5fd5599bd806112a0e" gracePeriod=30
Nov 03 17:50:53.350126 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:53.350100    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-8gsxp"
Nov 03 17:50:53.499529 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:53.499435    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c11a31f-ff69-4b63-b672-3d32e92ed503-kube-api-access-f9d8n\" (UniqueName: \"kubernetes.io/projected/8c11a31f-ff69-4b63-b672-3d32e92ed503-kube-api-access-f9d8n\") pod \"8c11a31f-ff69-4b63-b672-3d32e92ed503\" (UID: \"8c11a31f-ff69-4b63-b672-3d32e92ed503\") "
Nov 03 17:50:53.501733 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:53.501697    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c11a31f-ff69-4b63-b672-3d32e92ed503-kube-api-access-f9d8n" pod "8c11a31f-ff69-4b63-b672-3d32e92ed503" (UID: "8c11a31f-ff69-4b63-b672-3d32e92ed503"). InnerVolumeSpecName "kube-api-access-f9d8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:50:53.599991 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:53.599950    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9d8n\" (UniqueName: \"kubernetes.io/projected/8c11a31f-ff69-4b63-b672-3d32e92ed503-kube-api-access-f9d8n\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:50:54.116825 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:54.116790    5315 generic.go:356] "Generic (PLEG): container finished" podID="8c11a31f-ff69-4b63-b672-3d32e92ed503" containerID="7a728334b34e45444d847c455c65c80bc81e0970b4156f5fd5599bd806112a0e" exitCode=0
Nov 03 17:50:54.117289 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:54.116837    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-8gsxp"
Nov 03 17:50:54.117289 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:54.116873    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-8gsxp" event={"ID":"8c11a31f-ff69-4b63-b672-3d32e92ed503","Type":"ContainerDied","Data":"7a728334b34e45444d847c455c65c80bc81e0970b4156f5fd5599bd806112a0e"}
Nov 03 17:50:54.117289 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:54.116918    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-8gsxp" event={"ID":"8c11a31f-ff69-4b63-b672-3d32e92ed503","Type":"ContainerDied","Data":"f62a477627aad4925fe171a78e6f606036eed1fe579436f7d7a5d5c1eef9af32"}
Nov 03 17:50:54.117289 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:54.116940    5315 scope.go:122] "RemoveContainer" containerID="7a728334b34e45444d847c455c65c80bc81e0970b4156f5fd5599bd806112a0e"
Nov 03 17:50:54.125368 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:54.125346    5315 scope.go:122] "RemoveContainer" containerID="7a728334b34e45444d847c455c65c80bc81e0970b4156f5fd5599bd806112a0e"
Nov 03 17:50:54.125703 ip-10-0-128-204 kubenswrapper[5315]: E1103 17:50:54.125682    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a728334b34e45444d847c455c65c80bc81e0970b4156f5fd5599bd806112a0e\": container with ID starting with 7a728334b34e45444d847c455c65c80bc81e0970b4156f5fd5599bd806112a0e not found: ID does not exist" containerID="7a728334b34e45444d847c455c65c80bc81e0970b4156f5fd5599bd806112a0e"
Nov 03 17:50:54.125785 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:54.125709    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a728334b34e45444d847c455c65c80bc81e0970b4156f5fd5599bd806112a0e"} err="failed to get container status \"7a728334b34e45444d847c455c65c80bc81e0970b4156f5fd5599bd806112a0e\": rpc error: code = NotFound desc = could not find container \"7a728334b34e45444d847c455c65c80bc81e0970b4156f5fd5599bd806112a0e\": container with ID starting with 7a728334b34e45444d847c455c65c80bc81e0970b4156f5fd5599bd806112a0e not found: ID does not exist"
Nov 03 17:50:54.279954 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:54.279908    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-8gsxp"]
Nov 03 17:50:54.325823 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:54.325789    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-8gsxp"]
Nov 03 17:50:54.577968 ip-10-0-128-204 kubenswrapper[5315]: E1103 17:50:54.577934    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 17:50:55.101724 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:50:55.101682    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c11a31f-ff69-4b63-b672-3d32e92ed503" path="/var/lib/kubelet/pods/8c11a31f-ff69-4b63-b672-3d32e92ed503/volumes"
Nov 03 17:51:01.114013 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:01.113988    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-97p65_c29d200a-c917-46e5-9cf5-21f4d952e470/ovn-acl-logging/0.log"
Nov 03 17:51:01.114687 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:01.114662    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-97p65_c29d200a-c917-46e5-9cf5-21f4d952e470/ovn-acl-logging/0.log"
Nov 03 17:51:24.576393 ip-10-0-128-204 kubenswrapper[5315]: E1103 17:51:24.576345    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 17:51:50.810941 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:50.810906    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvhg2"]
Nov 03 17:51:50.811354 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:50.811231    5315 cpu_manager.go:414] "containerMap: removing container" podUID="8c11a31f-ff69-4b63-b672-3d32e92ed503" containerName="authorino"
Nov 03 17:51:50.811354 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:50.811245    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c11a31f-ff69-4b63-b672-3d32e92ed503" containerName="authorino"
Nov 03 17:51:50.811354 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:50.811296    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c11a31f-ff69-4b63-b672-3d32e92ed503" containerName="authorino"
Nov 03 17:51:50.813917 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:50.813901    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvhg2"
Nov 03 17:51:50.915077 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:50.915043    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvhg2"]
Nov 03 17:51:50.934404 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:50.934369    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8522e087-40fb-44bf-88e4-b8ce1ca03ec8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bvhg2\" (UID: \"8522e087-40fb-44bf-88e4-b8ce1ca03ec8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvhg2"
Nov 03 17:51:50.934570 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:50.934408    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8522e087-40fb-44bf-88e4-b8ce1ca03ec8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bvhg2\" (UID: \"8522e087-40fb-44bf-88e4-b8ce1ca03ec8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvhg2"
Nov 03 17:51:50.934570 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:50.934503    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8522e087-40fb-44bf-88e4-b8ce1ca03ec8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bvhg2\" (UID: \"8522e087-40fb-44bf-88e4-b8ce1ca03ec8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvhg2"
Nov 03 17:51:50.934570 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:50.934563    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8522e087-40fb-44bf-88e4-b8ce1ca03ec8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bvhg2\" (UID: \"8522e087-40fb-44bf-88e4-b8ce1ca03ec8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvhg2"
Nov 03 17:51:50.934706 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:50.934585    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8522e087-40fb-44bf-88e4-b8ce1ca03ec8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bvhg2\" (UID: \"8522e087-40fb-44bf-88e4-b8ce1ca03ec8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvhg2"
Nov 03 17:51:50.934706 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:50.934632    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8522e087-40fb-44bf-88e4-b8ce1ca03ec8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bvhg2\" (UID: \"8522e087-40fb-44bf-88e4-b8ce1ca03ec8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvhg2"
Nov 03 17:51:50.934706 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:50.934686    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9rzn\" (UniqueName: \"kubernetes.io/projected/8522e087-40fb-44bf-88e4-b8ce1ca03ec8-kube-api-access-f9rzn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bvhg2\" (UID: \"8522e087-40fb-44bf-88e4-b8ce1ca03ec8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvhg2"
Nov 03 17:51:51.035523 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:51.035485    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8522e087-40fb-44bf-88e4-b8ce1ca03ec8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bvhg2\" (UID: \"8522e087-40fb-44bf-88e4-b8ce1ca03ec8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvhg2"
Nov 03 17:51:51.035728 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:51.035532    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8522e087-40fb-44bf-88e4-b8ce1ca03ec8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bvhg2\" (UID: \"8522e087-40fb-44bf-88e4-b8ce1ca03ec8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvhg2"
Nov 03 17:51:51.035728 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:51.035550    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8522e087-40fb-44bf-88e4-b8ce1ca03ec8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bvhg2\" (UID: \"8522e087-40fb-44bf-88e4-b8ce1ca03ec8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvhg2"
Nov 03 17:51:51.035728 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:51.035581    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8522e087-40fb-44bf-88e4-b8ce1ca03ec8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bvhg2\" (UID: \"8522e087-40fb-44bf-88e4-b8ce1ca03ec8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvhg2"
Nov 03 17:51:51.035728 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:51.035605    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9rzn\" (UniqueName: \"kubernetes.io/projected/8522e087-40fb-44bf-88e4-b8ce1ca03ec8-kube-api-access-f9rzn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bvhg2\" (UID: \"8522e087-40fb-44bf-88e4-b8ce1ca03ec8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvhg2"
Nov 03 17:51:51.035728 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:51.035673    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8522e087-40fb-44bf-88e4-b8ce1ca03ec8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bvhg2\" (UID: \"8522e087-40fb-44bf-88e4-b8ce1ca03ec8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvhg2"
Nov 03 17:51:51.035728 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:51.035696    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8522e087-40fb-44bf-88e4-b8ce1ca03ec8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bvhg2\" (UID: \"8522e087-40fb-44bf-88e4-b8ce1ca03ec8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvhg2"
Nov 03 17:51:51.036330 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:51.036298    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8522e087-40fb-44bf-88e4-b8ce1ca03ec8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bvhg2\" (UID: \"8522e087-40fb-44bf-88e4-b8ce1ca03ec8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvhg2"
Nov 03 17:51:51.037936 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:51.037910    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8522e087-40fb-44bf-88e4-b8ce1ca03ec8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bvhg2\" (UID: \"8522e087-40fb-44bf-88e4-b8ce1ca03ec8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvhg2"
Nov 03 17:51:51.038061 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:51.038043    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8522e087-40fb-44bf-88e4-b8ce1ca03ec8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bvhg2\" (UID: \"8522e087-40fb-44bf-88e4-b8ce1ca03ec8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvhg2"
Nov 03 17:51:51.038295 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:51.038279    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8522e087-40fb-44bf-88e4-b8ce1ca03ec8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bvhg2\" (UID: \"8522e087-40fb-44bf-88e4-b8ce1ca03ec8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvhg2"
Nov 03 17:51:51.038408 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:51.038386    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8522e087-40fb-44bf-88e4-b8ce1ca03ec8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bvhg2\" (UID: \"8522e087-40fb-44bf-88e4-b8ce1ca03ec8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvhg2"
Nov 03 17:51:51.124527 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:51.124461    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9rzn\" (UniqueName: \"kubernetes.io/projected/8522e087-40fb-44bf-88e4-b8ce1ca03ec8-kube-api-access-f9rzn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bvhg2\" (UID: \"8522e087-40fb-44bf-88e4-b8ce1ca03ec8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvhg2"
Nov 03 17:51:51.133682 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:51.133660    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8522e087-40fb-44bf-88e4-b8ce1ca03ec8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bvhg2\" (UID: \"8522e087-40fb-44bf-88e4-b8ce1ca03ec8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvhg2"
Nov 03 17:51:51.425873 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:51.425786    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvhg2"
Nov 03 17:51:51.844732 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:51.844696    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvhg2"]
Nov 03 17:51:51.852897 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:51:51.852856    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8522e087_40fb_44bf_88e4_b8ce1ca03ec8.slice/crio-6dd377161a279a5513d72d77ef318d0abb1e28188de38c2c6418349cec52c148 WatchSource:0}: Error finding container 6dd377161a279a5513d72d77ef318d0abb1e28188de38c2c6418349cec52c148: Status 404 returned error can't find the container with id 6dd377161a279a5513d72d77ef318d0abb1e28188de38c2c6418349cec52c148
Nov 03 17:51:51.854945 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:51.854907    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 17:51:51.855050 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:51.854989    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 17:51:52.320254 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:52.320215    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvhg2" event={"ID":"8522e087-40fb-44bf-88e4-b8ce1ca03ec8","Type":"ContainerStarted","Data":"c2fee50afd87a186b46691b182858ba5c016a8c87a928ffff436b5b140e5aedb"}
Nov 03 17:51:52.320451 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:52.320295    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvhg2" event={"ID":"8522e087-40fb-44bf-88e4-b8ce1ca03ec8","Type":"ContainerStarted","Data":"6dd377161a279a5513d72d77ef318d0abb1e28188de38c2c6418349cec52c148"}
Nov 03 17:51:52.320451 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:52.320406    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvhg2"
Nov 03 17:51:52.321825 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:52.321800    5315 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-bvhg2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 17:51:52.321944 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:52.321843    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvhg2" podUID="8522e087-40fb-44bf-88e4-b8ce1ca03ec8" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 17:51:52.514443 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:52.514382    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvhg2" podStartSLOduration=2.514367639 podStartE2EDuration="2.514367639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:51:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:51:52.513760463 +0000 UTC m=+351.529958539" watchObservedRunningTime="2026-11-03 17:51:52.514367639 +0000 UTC m=+351.530565708"
Nov 03 17:51:53.324497 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:53.324469    5315 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-bvhg2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 17:51:53.324882 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:53.324513    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvhg2" podUID="8522e087-40fb-44bf-88e4-b8ce1ca03ec8" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 17:51:54.576460 ip-10-0-128-204 kubenswrapper[5315]: E1103 17:51:54.576422    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 17:51:56.325338 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:56.325303    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvhg2"
Nov 03 17:51:56.487265 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:56.487231    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-k67zm"]
Nov 03 17:51:56.487573 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:56.487531    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k67zm" podUID="cc1495e5-5a42-4d6a-b2ce-c32090fdca5c" containerName="discovery" containerID="cri-o://771a43056625f47d455411557369d862880a7e8872b19fb1a017ded92ad870b8" gracePeriod=30
Nov 03 17:51:56.733219 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:56.733191    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k67zm"
Nov 03 17:51:56.785373 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:56.785342    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc1495e5-5a42-4d6a-b2ce-c32090fdca5c-kube-api-access-vw7dl\" (UniqueName: \"kubernetes.io/projected/cc1495e5-5a42-4d6a-b2ce-c32090fdca5c-kube-api-access-vw7dl\") pod \"cc1495e5-5a42-4d6a-b2ce-c32090fdca5c\" (UID: \"cc1495e5-5a42-4d6a-b2ce-c32090fdca5c\") "
Nov 03 17:51:56.785373 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:56.785375    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc1495e5-5a42-4d6a-b2ce-c32090fdca5c-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cc1495e5-5a42-4d6a-b2ce-c32090fdca5c-istio-csr-ca-configmap\") pod \"cc1495e5-5a42-4d6a-b2ce-c32090fdca5c\" (UID: \"cc1495e5-5a42-4d6a-b2ce-c32090fdca5c\") "
Nov 03 17:51:56.785555 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:56.785412    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc1495e5-5a42-4d6a-b2ce-c32090fdca5c-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cc1495e5-5a42-4d6a-b2ce-c32090fdca5c-istio-kubeconfig\") pod \"cc1495e5-5a42-4d6a-b2ce-c32090fdca5c\" (UID: \"cc1495e5-5a42-4d6a-b2ce-c32090fdca5c\") "
Nov 03 17:51:56.785555 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:56.785449    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc1495e5-5a42-4d6a-b2ce-c32090fdca5c-cacerts\" (UniqueName: \"kubernetes.io/secret/cc1495e5-5a42-4d6a-b2ce-c32090fdca5c-cacerts\") pod \"cc1495e5-5a42-4d6a-b2ce-c32090fdca5c\" (UID: \"cc1495e5-5a42-4d6a-b2ce-c32090fdca5c\") "
Nov 03 17:51:56.785555 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:56.785472    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc1495e5-5a42-4d6a-b2ce-c32090fdca5c-istio-token\" (UniqueName: \"kubernetes.io/projected/cc1495e5-5a42-4d6a-b2ce-c32090fdca5c-istio-token\") pod \"cc1495e5-5a42-4d6a-b2ce-c32090fdca5c\" (UID: \"cc1495e5-5a42-4d6a-b2ce-c32090fdca5c\") "
Nov 03 17:51:56.785555 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:56.785491    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc1495e5-5a42-4d6a-b2ce-c32090fdca5c-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cc1495e5-5a42-4d6a-b2ce-c32090fdca5c-istio-csr-dns-cert\") pod \"cc1495e5-5a42-4d6a-b2ce-c32090fdca5c\" (UID: \"cc1495e5-5a42-4d6a-b2ce-c32090fdca5c\") "
Nov 03 17:51:56.785555 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:56.785528    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc1495e5-5a42-4d6a-b2ce-c32090fdca5c-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc1495e5-5a42-4d6a-b2ce-c32090fdca5c-local-certs\") pod \"cc1495e5-5a42-4d6a-b2ce-c32090fdca5c\" (UID: \"cc1495e5-5a42-4d6a-b2ce-c32090fdca5c\") "
Nov 03 17:51:56.786050 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:56.785998    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc1495e5-5a42-4d6a-b2ce-c32090fdca5c-istio-csr-ca-configmap" pod "cc1495e5-5a42-4d6a-b2ce-c32090fdca5c" (UID: "cc1495e5-5a42-4d6a-b2ce-c32090fdca5c"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:51:56.788349 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:56.788309    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc1495e5-5a42-4d6a-b2ce-c32090fdca5c-kube-api-access-vw7dl" pod "cc1495e5-5a42-4d6a-b2ce-c32090fdca5c" (UID: "cc1495e5-5a42-4d6a-b2ce-c32090fdca5c"). InnerVolumeSpecName "kube-api-access-vw7dl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:51:56.788468 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:56.788310    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc1495e5-5a42-4d6a-b2ce-c32090fdca5c-cacerts" pod "cc1495e5-5a42-4d6a-b2ce-c32090fdca5c" (UID: "cc1495e5-5a42-4d6a-b2ce-c32090fdca5c"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:51:56.793187 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:56.793164    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc1495e5-5a42-4d6a-b2ce-c32090fdca5c-istio-csr-dns-cert" pod "cc1495e5-5a42-4d6a-b2ce-c32090fdca5c" (UID: "cc1495e5-5a42-4d6a-b2ce-c32090fdca5c"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:51:56.793257 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:56.793181    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc1495e5-5a42-4d6a-b2ce-c32090fdca5c-local-certs" pod "cc1495e5-5a42-4d6a-b2ce-c32090fdca5c" (UID: "cc1495e5-5a42-4d6a-b2ce-c32090fdca5c"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:51:56.793257 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:56.793193    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc1495e5-5a42-4d6a-b2ce-c32090fdca5c-istio-kubeconfig" pod "cc1495e5-5a42-4d6a-b2ce-c32090fdca5c" (UID: "cc1495e5-5a42-4d6a-b2ce-c32090fdca5c"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:51:56.793257 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:56.793190    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc1495e5-5a42-4d6a-b2ce-c32090fdca5c-istio-token" pod "cc1495e5-5a42-4d6a-b2ce-c32090fdca5c" (UID: "cc1495e5-5a42-4d6a-b2ce-c32090fdca5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:51:56.886677 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:56.886584    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw7dl\" (UniqueName: \"kubernetes.io/projected/cc1495e5-5a42-4d6a-b2ce-c32090fdca5c-kube-api-access-vw7dl\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:51:56.886677 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:56.886647    5315 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cc1495e5-5a42-4d6a-b2ce-c32090fdca5c-istio-csr-ca-configmap\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:51:56.886677 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:56.886662    5315 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cc1495e5-5a42-4d6a-b2ce-c32090fdca5c-istio-kubeconfig\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:51:56.886677 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:56.886672    5315 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cc1495e5-5a42-4d6a-b2ce-c32090fdca5c-cacerts\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:51:56.886677 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:56.886682    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc1495e5-5a42-4d6a-b2ce-c32090fdca5c-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:51:56.886942 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:56.886691    5315 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cc1495e5-5a42-4d6a-b2ce-c32090fdca5c-istio-csr-dns-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:51:56.886942 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:56.886699    5315 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc1495e5-5a42-4d6a-b2ce-c32090fdca5c-local-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:51:57.340729 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:57.340691    5315 generic.go:356] "Generic (PLEG): container finished" podID="cc1495e5-5a42-4d6a-b2ce-c32090fdca5c" containerID="771a43056625f47d455411557369d862880a7e8872b19fb1a017ded92ad870b8" exitCode=0
Nov 03 17:51:57.341126 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:57.340760    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k67zm" event={"ID":"cc1495e5-5a42-4d6a-b2ce-c32090fdca5c","Type":"ContainerDied","Data":"771a43056625f47d455411557369d862880a7e8872b19fb1a017ded92ad870b8"}
Nov 03 17:51:57.341126 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:57.340803    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k67zm" event={"ID":"cc1495e5-5a42-4d6a-b2ce-c32090fdca5c","Type":"ContainerDied","Data":"119a66783219c465865b5c570bd68762bbd662e973321177a16dbad1f703b948"}
Nov 03 17:51:57.341126 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:57.340822    5315 scope.go:122] "RemoveContainer" containerID="771a43056625f47d455411557369d862880a7e8872b19fb1a017ded92ad870b8"
Nov 03 17:51:57.341126 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:57.340766    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k67zm"
Nov 03 17:51:57.348624 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:57.348590    5315 scope.go:122] "RemoveContainer" containerID="771a43056625f47d455411557369d862880a7e8872b19fb1a017ded92ad870b8"
Nov 03 17:51:57.348900 ip-10-0-128-204 kubenswrapper[5315]: E1103 17:51:57.348881    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"771a43056625f47d455411557369d862880a7e8872b19fb1a017ded92ad870b8\": container with ID starting with 771a43056625f47d455411557369d862880a7e8872b19fb1a017ded92ad870b8 not found: ID does not exist" containerID="771a43056625f47d455411557369d862880a7e8872b19fb1a017ded92ad870b8"
Nov 03 17:51:57.348963 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:57.348914    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"771a43056625f47d455411557369d862880a7e8872b19fb1a017ded92ad870b8"} err="failed to get container status \"771a43056625f47d455411557369d862880a7e8872b19fb1a017ded92ad870b8\": rpc error: code = NotFound desc = could not find container \"771a43056625f47d455411557369d862880a7e8872b19fb1a017ded92ad870b8\": container with ID starting with 771a43056625f47d455411557369d862880a7e8872b19fb1a017ded92ad870b8 not found: ID does not exist"
Nov 03 17:51:57.494471 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:57.494438    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-k67zm"]
Nov 03 17:51:57.540232 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:57.540196    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-k67zm"]
Nov 03 17:51:59.102199 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:51:59.102164    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc1495e5-5a42-4d6a-b2ce-c32090fdca5c" path="/var/lib/kubelet/pods/cc1495e5-5a42-4d6a-b2ce-c32090fdca5c/volumes"
Nov 03 17:52:01.455784 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:01.455593    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-zc4hj"]
Nov 03 17:52:01.456241 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:01.456031    5315 cpu_manager.go:414] "containerMap: removing container" podUID="cc1495e5-5a42-4d6a-b2ce-c32090fdca5c" containerName="discovery"
Nov 03 17:52:01.456241 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:01.456050    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc1495e5-5a42-4d6a-b2ce-c32090fdca5c" containerName="discovery"
Nov 03 17:52:01.456241 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:01.456103    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc1495e5-5a42-4d6a-b2ce-c32090fdca5c" containerName="discovery"
Nov 03 17:52:01.460647 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:01.460603    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-zc4hj"
Nov 03 17:52:01.509547 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:01.509497    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 03 17:52:01.523034 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:01.522998    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlxj7\" (UniqueName: \"kubernetes.io/projected/e9f3f2e4-f79e-4d99-b782-b2d2631fe3e2-kube-api-access-nlxj7\") pod \"authorino-6f6944bcdd-zc4hj\" (UID: \"e9f3f2e4-f79e-4d99-b782-b2d2631fe3e2\") " pod="kuadrant-system/authorino-6f6944bcdd-zc4hj"
Nov 03 17:52:01.555639 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:01.555602    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-4hst9\""
Nov 03 17:52:01.555783 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:01.555763    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 03 17:52:01.556340 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:01.556319    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-zc4hj"]
Nov 03 17:52:01.623868 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:01.623828    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlxj7\" (UniqueName: \"kubernetes.io/projected/e9f3f2e4-f79e-4d99-b782-b2d2631fe3e2-kube-api-access-nlxj7\") pod \"authorino-6f6944bcdd-zc4hj\" (UID: \"e9f3f2e4-f79e-4d99-b782-b2d2631fe3e2\") " pod="kuadrant-system/authorino-6f6944bcdd-zc4hj"
Nov 03 17:52:01.683854 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:01.683820    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlxj7\" (UniqueName: \"kubernetes.io/projected/e9f3f2e4-f79e-4d99-b782-b2d2631fe3e2-kube-api-access-nlxj7\") pod \"authorino-6f6944bcdd-zc4hj\" (UID: \"e9f3f2e4-f79e-4d99-b782-b2d2631fe3e2\") " pod="kuadrant-system/authorino-6f6944bcdd-zc4hj"
Nov 03 17:52:01.713676 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:01.713580    5315 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-k67zm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.13:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:52:01.713839 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:01.713674    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-k67zm" podUID="cc1495e5-5a42-4d6a-b2ce-c32090fdca5c" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.13:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:52:01.783906 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:01.783867    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-zc4hj"
Nov 03 17:52:01.851237 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:01.851158    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-zc4hj"]
Nov 03 17:52:01.876601 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:01.876563    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6ccd9d464-kg7mv"]
Nov 03 17:52:01.880060 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:01.880031    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6ccd9d464-kg7mv"
Nov 03 17:52:01.935127 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:01.935084    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6ccd9d464-kg7mv"]
Nov 03 17:52:02.026942 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:02.026909    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6ghw\" (UniqueName: \"kubernetes.io/projected/7d4e5db7-9d93-4e55-898e-ebcd28a5ac1e-kube-api-access-r6ghw\") pod \"authorino-6ccd9d464-kg7mv\" (UID: \"7d4e5db7-9d93-4e55-898e-ebcd28a5ac1e\") " pod="kuadrant-system/authorino-6ccd9d464-kg7mv"
Nov 03 17:52:02.127994 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:02.127957    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6ghw\" (UniqueName: \"kubernetes.io/projected/7d4e5db7-9d93-4e55-898e-ebcd28a5ac1e-kube-api-access-r6ghw\") pod \"authorino-6ccd9d464-kg7mv\" (UID: \"7d4e5db7-9d93-4e55-898e-ebcd28a5ac1e\") " pod="kuadrant-system/authorino-6ccd9d464-kg7mv"
Nov 03 17:52:02.147502 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:02.147469    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-zc4hj"]
Nov 03 17:52:02.154802 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:52:02.154771    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9f3f2e4_f79e_4d99_b782_b2d2631fe3e2.slice/crio-f777ee814b5135e8d712ba5d3243372751b0f0fa5f220f5ab40ca94e7c8f74ef WatchSource:0}: Error finding container f777ee814b5135e8d712ba5d3243372751b0f0fa5f220f5ab40ca94e7c8f74ef: Status 404 returned error can't find the container with id f777ee814b5135e8d712ba5d3243372751b0f0fa5f220f5ab40ca94e7c8f74ef
Nov 03 17:52:02.185750 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:02.185720    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6ghw\" (UniqueName: \"kubernetes.io/projected/7d4e5db7-9d93-4e55-898e-ebcd28a5ac1e-kube-api-access-r6ghw\") pod \"authorino-6ccd9d464-kg7mv\" (UID: \"7d4e5db7-9d93-4e55-898e-ebcd28a5ac1e\") " pod="kuadrant-system/authorino-6ccd9d464-kg7mv"
Nov 03 17:52:02.191456 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:02.191429    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6ccd9d464-kg7mv"
Nov 03 17:52:02.289639 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:02.289528    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6ccd9d464-kg7mv"]
Nov 03 17:52:02.328391 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:02.328361    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-865f494c86-pmhmr"]
Nov 03 17:52:02.331802 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:02.331778    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-865f494c86-pmhmr"
Nov 03 17:52:02.359435 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:02.359398    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-zc4hj" event={"ID":"e9f3f2e4-f79e-4d99-b782-b2d2631fe3e2","Type":"ContainerStarted","Data":"f777ee814b5135e8d712ba5d3243372751b0f0fa5f220f5ab40ca94e7c8f74ef"}
Nov 03 17:52:02.386694 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:02.386657    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-865f494c86-pmhmr"]
Nov 03 17:52:02.427823 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:02.427793    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Nov 03 17:52:02.431146 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:02.431116    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f53fdcc8-09c2-4c65-ae5b-c32cc3397b01-tls-cert\") pod \"authorino-865f494c86-pmhmr\" (UID: \"f53fdcc8-09c2-4c65-ae5b-c32cc3397b01\") " pod="kuadrant-system/authorino-865f494c86-pmhmr"
Nov 03 17:52:02.431285 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:02.431158    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x2tn\" (UniqueName: \"kubernetes.io/projected/f53fdcc8-09c2-4c65-ae5b-c32cc3397b01-kube-api-access-6x2tn\") pod \"authorino-865f494c86-pmhmr\" (UID: \"f53fdcc8-09c2-4c65-ae5b-c32cc3397b01\") " pod="kuadrant-system/authorino-865f494c86-pmhmr"
Nov 03 17:52:02.532204 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:02.532169    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f53fdcc8-09c2-4c65-ae5b-c32cc3397b01-tls-cert\") pod \"authorino-865f494c86-pmhmr\" (UID: \"f53fdcc8-09c2-4c65-ae5b-c32cc3397b01\") " pod="kuadrant-system/authorino-865f494c86-pmhmr"
Nov 03 17:52:02.532695 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:02.532215    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x2tn\" (UniqueName: \"kubernetes.io/projected/f53fdcc8-09c2-4c65-ae5b-c32cc3397b01-kube-api-access-6x2tn\") pod \"authorino-865f494c86-pmhmr\" (UID: \"f53fdcc8-09c2-4c65-ae5b-c32cc3397b01\") " pod="kuadrant-system/authorino-865f494c86-pmhmr"
Nov 03 17:52:02.534544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:02.534525    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f53fdcc8-09c2-4c65-ae5b-c32cc3397b01-tls-cert\") pod \"authorino-865f494c86-pmhmr\" (UID: \"f53fdcc8-09c2-4c65-ae5b-c32cc3397b01\") " pod="kuadrant-system/authorino-865f494c86-pmhmr"
Nov 03 17:52:02.559371 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:02.559285    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6ccd9d464-kg7mv"]
Nov 03 17:52:02.566441 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:52:02.566409    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d4e5db7_9d93_4e55_898e_ebcd28a5ac1e.slice/crio-89712a34271d04b8b4e46f5e061e5dcd69c3a18fe5e90d5296152088b08d8cbf WatchSource:0}: Error finding container 89712a34271d04b8b4e46f5e061e5dcd69c3a18fe5e90d5296152088b08d8cbf: Status 404 returned error can't find the container with id 89712a34271d04b8b4e46f5e061e5dcd69c3a18fe5e90d5296152088b08d8cbf
Nov 03 17:52:02.596215 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:02.596187    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x2tn\" (UniqueName: \"kubernetes.io/projected/f53fdcc8-09c2-4c65-ae5b-c32cc3397b01-kube-api-access-6x2tn\") pod \"authorino-865f494c86-pmhmr\" (UID: \"f53fdcc8-09c2-4c65-ae5b-c32cc3397b01\") " pod="kuadrant-system/authorino-865f494c86-pmhmr"
Nov 03 17:52:02.643122 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:02.643086    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-865f494c86-pmhmr"
Nov 03 17:52:03.020586 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:03.020501    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-865f494c86-pmhmr"]
Nov 03 17:52:03.027558 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:52:03.027527    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf53fdcc8_09c2_4c65_ae5b_c32cc3397b01.slice/crio-8897372951f1b862688be51ba101c756720aa2fd94559bbececcca43fbf77773 WatchSource:0}: Error finding container 8897372951f1b862688be51ba101c756720aa2fd94559bbececcca43fbf77773: Status 404 returned error can't find the container with id 8897372951f1b862688be51ba101c756720aa2fd94559bbececcca43fbf77773
Nov 03 17:52:03.363563 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:03.363516    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-865f494c86-pmhmr" event={"ID":"f53fdcc8-09c2-4c65-ae5b-c32cc3397b01","Type":"ContainerStarted","Data":"8897372951f1b862688be51ba101c756720aa2fd94559bbececcca43fbf77773"}
Nov 03 17:52:03.364709 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:03.364685    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6ccd9d464-kg7mv" event={"ID":"7d4e5db7-9d93-4e55-898e-ebcd28a5ac1e","Type":"ContainerStarted","Data":"89712a34271d04b8b4e46f5e061e5dcd69c3a18fe5e90d5296152088b08d8cbf"}
Nov 03 17:52:04.373454 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:04.373416    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-865f494c86-pmhmr" event={"ID":"f53fdcc8-09c2-4c65-ae5b-c32cc3397b01","Type":"ContainerStarted","Data":"1a299ca3550824c634716c211736913904729a63cacdfe63e83d52114f32b108"}
Nov 03 17:52:04.376531 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:04.376478    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6ccd9d464-kg7mv" event={"ID":"7d4e5db7-9d93-4e55-898e-ebcd28a5ac1e","Type":"ContainerStarted","Data":"26e71f3b3b71c28ff9f1c940353d40b221c1105b4de20b293215f5e1e6f90c10"}
Nov 03 17:52:04.376709 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:04.376601    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6ccd9d464-kg7mv" podUID="7d4e5db7-9d93-4e55-898e-ebcd28a5ac1e" containerName="authorino" containerID="cri-o://26e71f3b3b71c28ff9f1c940353d40b221c1105b4de20b293215f5e1e6f90c10" gracePeriod=30
Nov 03 17:52:04.590157 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:04.590100    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6ccd9d464-kg7mv" podStartSLOduration=2.019861144 podStartE2EDuration="3.590086427s" totalImagesPullingTime="1.570225283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:52:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:52:04.59001525 +0000 UTC m=+363.606213321" watchObservedRunningTime="2026-11-03 17:52:04.590086427 +0000 UTC m=+363.606284495"
Nov 03 17:52:04.590412 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:04.590184    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-865f494c86-pmhmr" podStartSLOduration=1.337754619 podStartE2EDuration="2.590178587s" totalImagesPullingTime="1.252423968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:52:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:52:04.481539593 +0000 UTC m=+363.497737666" watchObservedRunningTime="2026-11-03 17:52:04.590178587 +0000 UTC m=+363.606376653"
Nov 03 17:52:04.618517 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:04.618492    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6ccd9d464-kg7mv"
Nov 03 17:52:04.751531 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:04.751450    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d4e5db7-9d93-4e55-898e-ebcd28a5ac1e-kube-api-access-r6ghw\" (UniqueName: \"kubernetes.io/projected/7d4e5db7-9d93-4e55-898e-ebcd28a5ac1e-kube-api-access-r6ghw\") pod \"7d4e5db7-9d93-4e55-898e-ebcd28a5ac1e\" (UID: \"7d4e5db7-9d93-4e55-898e-ebcd28a5ac1e\") "
Nov 03 17:52:04.753494 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:04.753469    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d4e5db7-9d93-4e55-898e-ebcd28a5ac1e-kube-api-access-r6ghw" pod "7d4e5db7-9d93-4e55-898e-ebcd28a5ac1e" (UID: "7d4e5db7-9d93-4e55-898e-ebcd28a5ac1e"). InnerVolumeSpecName "kube-api-access-r6ghw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:52:04.853022 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:04.852988    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6ghw\" (UniqueName: \"kubernetes.io/projected/7d4e5db7-9d93-4e55-898e-ebcd28a5ac1e-kube-api-access-r6ghw\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:52:05.380716 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:05.380622    5315 generic.go:356] "Generic (PLEG): container finished" podID="7d4e5db7-9d93-4e55-898e-ebcd28a5ac1e" containerID="26e71f3b3b71c28ff9f1c940353d40b221c1105b4de20b293215f5e1e6f90c10" exitCode=0
Nov 03 17:52:05.380716 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:05.380667    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6ccd9d464-kg7mv"
Nov 03 17:52:05.381179 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:05.380709    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6ccd9d464-kg7mv" event={"ID":"7d4e5db7-9d93-4e55-898e-ebcd28a5ac1e","Type":"ContainerDied","Data":"26e71f3b3b71c28ff9f1c940353d40b221c1105b4de20b293215f5e1e6f90c10"}
Nov 03 17:52:05.381179 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:05.380749    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6ccd9d464-kg7mv" event={"ID":"7d4e5db7-9d93-4e55-898e-ebcd28a5ac1e","Type":"ContainerDied","Data":"89712a34271d04b8b4e46f5e061e5dcd69c3a18fe5e90d5296152088b08d8cbf"}
Nov 03 17:52:05.381179 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:05.380766    5315 scope.go:122] "RemoveContainer" containerID="26e71f3b3b71c28ff9f1c940353d40b221c1105b4de20b293215f5e1e6f90c10"
Nov 03 17:52:05.388431 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:05.388411    5315 scope.go:122] "RemoveContainer" containerID="26e71f3b3b71c28ff9f1c940353d40b221c1105b4de20b293215f5e1e6f90c10"
Nov 03 17:52:05.388727 ip-10-0-128-204 kubenswrapper[5315]: E1103 17:52:05.388708    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26e71f3b3b71c28ff9f1c940353d40b221c1105b4de20b293215f5e1e6f90c10\": container with ID starting with 26e71f3b3b71c28ff9f1c940353d40b221c1105b4de20b293215f5e1e6f90c10 not found: ID does not exist" containerID="26e71f3b3b71c28ff9f1c940353d40b221c1105b4de20b293215f5e1e6f90c10"
Nov 03 17:52:05.388787 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:05.388736    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26e71f3b3b71c28ff9f1c940353d40b221c1105b4de20b293215f5e1e6f90c10"} err="failed to get container status \"26e71f3b3b71c28ff9f1c940353d40b221c1105b4de20b293215f5e1e6f90c10\": rpc error: code = NotFound desc = could not find container \"26e71f3b3b71c28ff9f1c940353d40b221c1105b4de20b293215f5e1e6f90c10\": container with ID starting with 26e71f3b3b71c28ff9f1c940353d40b221c1105b4de20b293215f5e1e6f90c10 not found: ID does not exist"
Nov 03 17:52:05.532981 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:05.532945    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6ccd9d464-kg7mv"]
Nov 03 17:52:05.578598 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:05.578564    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6ccd9d464-kg7mv"]
Nov 03 17:52:07.101839 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:07.101797    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d4e5db7-9d93-4e55-898e-ebcd28a5ac1e" path="/var/lib/kubelet/pods/7d4e5db7-9d93-4e55-898e-ebcd28a5ac1e/volumes"
Nov 03 17:52:08.394623 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:08.394562    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-zc4hj" event={"ID":"e9f3f2e4-f79e-4d99-b782-b2d2631fe3e2","Type":"ContainerStarted","Data":"bd1a852022b9a404ff46bd32f70f45c352e4e3bafe4a5f51a79a3728a7b2edab"}
Nov 03 17:52:08.394623 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:08.394583    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-zc4hj" podUID="e9f3f2e4-f79e-4d99-b782-b2d2631fe3e2" containerName="authorino" containerID="cri-o://bd1a852022b9a404ff46bd32f70f45c352e4e3bafe4a5f51a79a3728a7b2edab" gracePeriod=30
Nov 03 17:52:08.505387 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:08.505337    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-zc4hj" podStartSLOduration=1.964668573 podStartE2EDuration="7.505306945s" totalImagesPullingTime="5.540638372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:52:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:52:08.504389107 +0000 UTC m=+367.520587176" watchObservedRunningTime="2026-11-03 17:52:08.505306945 +0000 UTC m=+367.521505014"
Nov 03 17:52:08.646700 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:08.646639    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-zc4hj"
Nov 03 17:52:08.784108 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:08.784070    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9f3f2e4-f79e-4d99-b782-b2d2631fe3e2-kube-api-access-nlxj7\" (UniqueName: \"kubernetes.io/projected/e9f3f2e4-f79e-4d99-b782-b2d2631fe3e2-kube-api-access-nlxj7\") pod \"e9f3f2e4-f79e-4d99-b782-b2d2631fe3e2\" (UID: \"e9f3f2e4-f79e-4d99-b782-b2d2631fe3e2\") "
Nov 03 17:52:08.786102 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:08.786067    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9f3f2e4-f79e-4d99-b782-b2d2631fe3e2-kube-api-access-nlxj7" pod "e9f3f2e4-f79e-4d99-b782-b2d2631fe3e2" (UID: "e9f3f2e4-f79e-4d99-b782-b2d2631fe3e2"). InnerVolumeSpecName "kube-api-access-nlxj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:52:08.884882 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:08.884845    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlxj7\" (UniqueName: \"kubernetes.io/projected/e9f3f2e4-f79e-4d99-b782-b2d2631fe3e2-kube-api-access-nlxj7\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:52:09.399064 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:09.398975    5315 generic.go:356] "Generic (PLEG): container finished" podID="e9f3f2e4-f79e-4d99-b782-b2d2631fe3e2" containerID="bd1a852022b9a404ff46bd32f70f45c352e4e3bafe4a5f51a79a3728a7b2edab" exitCode=0
Nov 03 17:52:09.399064 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:09.399023    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-zc4hj"
Nov 03 17:52:09.399537 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:09.399072    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-zc4hj" event={"ID":"e9f3f2e4-f79e-4d99-b782-b2d2631fe3e2","Type":"ContainerDied","Data":"bd1a852022b9a404ff46bd32f70f45c352e4e3bafe4a5f51a79a3728a7b2edab"}
Nov 03 17:52:09.399537 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:09.399122    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-zc4hj" event={"ID":"e9f3f2e4-f79e-4d99-b782-b2d2631fe3e2","Type":"ContainerDied","Data":"f777ee814b5135e8d712ba5d3243372751b0f0fa5f220f5ab40ca94e7c8f74ef"}
Nov 03 17:52:09.399537 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:09.399143    5315 scope.go:122] "RemoveContainer" containerID="bd1a852022b9a404ff46bd32f70f45c352e4e3bafe4a5f51a79a3728a7b2edab"
Nov 03 17:52:09.406945 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:09.406925    5315 scope.go:122] "RemoveContainer" containerID="bd1a852022b9a404ff46bd32f70f45c352e4e3bafe4a5f51a79a3728a7b2edab"
Nov 03 17:52:09.407241 ip-10-0-128-204 kubenswrapper[5315]: E1103 17:52:09.407219    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd1a852022b9a404ff46bd32f70f45c352e4e3bafe4a5f51a79a3728a7b2edab\": container with ID starting with bd1a852022b9a404ff46bd32f70f45c352e4e3bafe4a5f51a79a3728a7b2edab not found: ID does not exist" containerID="bd1a852022b9a404ff46bd32f70f45c352e4e3bafe4a5f51a79a3728a7b2edab"
Nov 03 17:52:09.407298 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:09.407249    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd1a852022b9a404ff46bd32f70f45c352e4e3bafe4a5f51a79a3728a7b2edab"} err="failed to get container status \"bd1a852022b9a404ff46bd32f70f45c352e4e3bafe4a5f51a79a3728a7b2edab\": rpc error: code = NotFound desc = could not find container \"bd1a852022b9a404ff46bd32f70f45c352e4e3bafe4a5f51a79a3728a7b2edab\": container with ID starting with bd1a852022b9a404ff46bd32f70f45c352e4e3bafe4a5f51a79a3728a7b2edab not found: ID does not exist"
Nov 03 17:52:09.550238 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:09.550196    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-zc4hj"]
Nov 03 17:52:09.596012 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:09.595980    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-zc4hj"]
Nov 03 17:52:11.101342 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:52:11.101310    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9f3f2e4-f79e-4d99-b782-b2d2631fe3e2" path="/var/lib/kubelet/pods/e9f3f2e4-f79e-4d99-b782-b2d2631fe3e2/volumes"
Nov 03 17:52:24.576071 ip-10-0-128-204 kubenswrapper[5315]: E1103 17:52:24.576029    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 17:52:54.576003 ip-10-0-128-204 kubenswrapper[5315]: E1103 17:52:54.575968    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 17:53:24.575385 ip-10-0-128-204 kubenswrapper[5315]: E1103 17:53:24.575341    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 17:53:54.577413 ip-10-0-128-204 kubenswrapper[5315]: E1103 17:53:54.577370    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 17:54:01.364390 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:01.364355    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx"]
Nov 03 17:54:01.364876 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:01.364651    5315 cpu_manager.go:414] "containerMap: removing container" podUID="e9f3f2e4-f79e-4d99-b782-b2d2631fe3e2" containerName="authorino"
Nov 03 17:54:01.364876 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:01.364663    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9f3f2e4-f79e-4d99-b782-b2d2631fe3e2" containerName="authorino"
Nov 03 17:54:01.364876 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:01.364679    5315 cpu_manager.go:414] "containerMap: removing container" podUID="7d4e5db7-9d93-4e55-898e-ebcd28a5ac1e" containerName="authorino"
Nov 03 17:54:01.364876 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:01.364684    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d4e5db7-9d93-4e55-898e-ebcd28a5ac1e" containerName="authorino"
Nov 03 17:54:01.364876 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:01.364730    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9f3f2e4-f79e-4d99-b782-b2d2631fe3e2" containerName="authorino"
Nov 03 17:54:01.364876 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:01.364739    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d4e5db7-9d93-4e55-898e-ebcd28a5ac1e" containerName="authorino"
Nov 03 17:54:01.367538 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:01.367518    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx"
Nov 03 17:54:01.450215 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:01.450186    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/626f43f8-df88-48d5-9f11-ca75310a0fc2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx\" (UID: \"626f43f8-df88-48d5-9f11-ca75310a0fc2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx"
Nov 03 17:54:01.450371 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:01.450221    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/626f43f8-df88-48d5-9f11-ca75310a0fc2-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx\" (UID: \"626f43f8-df88-48d5-9f11-ca75310a0fc2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx"
Nov 03 17:54:01.450371 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:01.450248    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtpd7\" (UniqueName: \"kubernetes.io/projected/626f43f8-df88-48d5-9f11-ca75310a0fc2-kube-api-access-vtpd7\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx\" (UID: \"626f43f8-df88-48d5-9f11-ca75310a0fc2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx"
Nov 03 17:54:01.450453 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:01.450383    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/626f43f8-df88-48d5-9f11-ca75310a0fc2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx\" (UID: \"626f43f8-df88-48d5-9f11-ca75310a0fc2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx"
Nov 03 17:54:01.450453 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:01.450407    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/626f43f8-df88-48d5-9f11-ca75310a0fc2-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx\" (UID: \"626f43f8-df88-48d5-9f11-ca75310a0fc2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx"
Nov 03 17:54:01.450453 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:01.450427    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/626f43f8-df88-48d5-9f11-ca75310a0fc2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx\" (UID: \"626f43f8-df88-48d5-9f11-ca75310a0fc2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx"
Nov 03 17:54:01.462147 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:01.462122    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Nov 03 17:54:01.462229 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:01.462170    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 03 17:54:01.462229 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:01.462204    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-x9qtm\""
Nov 03 17:54:01.462345 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:01.462333    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 03 17:54:01.463125 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:01.463109    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx"]
Nov 03 17:54:01.551554 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:01.551520    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/626f43f8-df88-48d5-9f11-ca75310a0fc2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx\" (UID: \"626f43f8-df88-48d5-9f11-ca75310a0fc2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx"
Nov 03 17:54:01.551554 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:01.551555    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/626f43f8-df88-48d5-9f11-ca75310a0fc2-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx\" (UID: \"626f43f8-df88-48d5-9f11-ca75310a0fc2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx"
Nov 03 17:54:01.551775 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:01.551576    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/626f43f8-df88-48d5-9f11-ca75310a0fc2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx\" (UID: \"626f43f8-df88-48d5-9f11-ca75310a0fc2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx"
Nov 03 17:54:01.551775 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:01.551693    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/626f43f8-df88-48d5-9f11-ca75310a0fc2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx\" (UID: \"626f43f8-df88-48d5-9f11-ca75310a0fc2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx"
Nov 03 17:54:01.551775 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:01.551738    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/626f43f8-df88-48d5-9f11-ca75310a0fc2-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx\" (UID: \"626f43f8-df88-48d5-9f11-ca75310a0fc2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx"
Nov 03 17:54:01.551899 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:01.551789    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtpd7\" (UniqueName: \"kubernetes.io/projected/626f43f8-df88-48d5-9f11-ca75310a0fc2-kube-api-access-vtpd7\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx\" (UID: \"626f43f8-df88-48d5-9f11-ca75310a0fc2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx"
Nov 03 17:54:01.552049 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:01.552026    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/626f43f8-df88-48d5-9f11-ca75310a0fc2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx\" (UID: \"626f43f8-df88-48d5-9f11-ca75310a0fc2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx"
Nov 03 17:54:01.552175 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:01.552156    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/626f43f8-df88-48d5-9f11-ca75310a0fc2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx\" (UID: \"626f43f8-df88-48d5-9f11-ca75310a0fc2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx"
Nov 03 17:54:01.552242 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:01.552217    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/626f43f8-df88-48d5-9f11-ca75310a0fc2-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx\" (UID: \"626f43f8-df88-48d5-9f11-ca75310a0fc2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx"
Nov 03 17:54:01.553816 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:01.553798    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/626f43f8-df88-48d5-9f11-ca75310a0fc2-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx\" (UID: \"626f43f8-df88-48d5-9f11-ca75310a0fc2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx"
Nov 03 17:54:01.554073 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:01.554056    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/626f43f8-df88-48d5-9f11-ca75310a0fc2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx\" (UID: \"626f43f8-df88-48d5-9f11-ca75310a0fc2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx"
Nov 03 17:54:01.609552 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:01.609512    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtpd7\" (UniqueName: \"kubernetes.io/projected/626f43f8-df88-48d5-9f11-ca75310a0fc2-kube-api-access-vtpd7\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx\" (UID: \"626f43f8-df88-48d5-9f11-ca75310a0fc2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx"
Nov 03 17:54:01.679755 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:01.679668    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx"
Nov 03 17:54:02.047474 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:02.047441    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx"]
Nov 03 17:54:02.055180 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:54:02.055146    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod626f43f8_df88_48d5_9f11_ca75310a0fc2.slice/crio-0f2a23f8fadaad16be1fd2fca79e43222aee02e8f5e8cc7e2291880a8c9eb8ad WatchSource:0}: Error finding container 0f2a23f8fadaad16be1fd2fca79e43222aee02e8f5e8cc7e2291880a8c9eb8ad: Status 404 returned error can't find the container with id 0f2a23f8fadaad16be1fd2fca79e43222aee02e8f5e8cc7e2291880a8c9eb8ad
Nov 03 17:54:02.056673 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:02.056647    5315 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 17:54:02.797733 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:02.797688    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx" event={"ID":"626f43f8-df88-48d5-9f11-ca75310a0fc2","Type":"ContainerStarted","Data":"0f2a23f8fadaad16be1fd2fca79e43222aee02e8f5e8cc7e2291880a8c9eb8ad"}
Nov 03 17:54:04.806969 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:04.806932    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx" event={"ID":"626f43f8-df88-48d5-9f11-ca75310a0fc2","Type":"ContainerStarted","Data":"7041f65ea8bcf27a41da30b0eaa54784bc773405f7f6034c7be0e96193eefb8f"}
Nov 03 17:54:04.917219 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:04.917163    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx" podStartSLOduration=1.753765573 podStartE2EDuration="3.917150266s" totalImagesPullingTime="2.163384693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:54:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:54:04.916291917 +0000 UTC m=+483.932489986" watchObservedRunningTime="2026-11-03 17:54:04.917150266 +0000 UTC m=+483.933348325"
Nov 03 17:54:05.679803 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:05.679764    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx"
Nov 03 17:54:11.680548 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:11.680501    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx"
Nov 03 17:54:11.693589 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:11.693555    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx"
Nov 03 17:54:11.844186 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:11.844154    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx"
Nov 03 17:54:12.524793 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:12.524759    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-7nllb"]
Nov 03 17:54:12.528056 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:12.528037    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7nllb"
Nov 03 17:54:12.599727 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:12.599698    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Nov 03 17:54:12.623354 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:12.623320    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-7nllb"]
Nov 03 17:54:12.641920 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:12.641889    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cc7a7a6-2c8d-426e-840c-3ef78e44e86a-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7nllb\" (UID: \"4cc7a7a6-2c8d-426e-840c-3ef78e44e86a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7nllb"
Nov 03 17:54:12.641920 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:12.641921    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cc7a7a6-2c8d-426e-840c-3ef78e44e86a-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7nllb\" (UID: \"4cc7a7a6-2c8d-426e-840c-3ef78e44e86a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7nllb"
Nov 03 17:54:12.642137 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:12.641953    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cc7a7a6-2c8d-426e-840c-3ef78e44e86a-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7nllb\" (UID: \"4cc7a7a6-2c8d-426e-840c-3ef78e44e86a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7nllb"
Nov 03 17:54:12.642137 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:12.642019    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cc7a7a6-2c8d-426e-840c-3ef78e44e86a-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7nllb\" (UID: \"4cc7a7a6-2c8d-426e-840c-3ef78e44e86a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7nllb"
Nov 03 17:54:12.642137 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:12.642095    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cc7a7a6-2c8d-426e-840c-3ef78e44e86a-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7nllb\" (UID: \"4cc7a7a6-2c8d-426e-840c-3ef78e44e86a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7nllb"
Nov 03 17:54:12.642268 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:12.642194    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nqn6\" (UniqueName: \"kubernetes.io/projected/4cc7a7a6-2c8d-426e-840c-3ef78e44e86a-kube-api-access-5nqn6\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7nllb\" (UID: \"4cc7a7a6-2c8d-426e-840c-3ef78e44e86a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7nllb"
Nov 03 17:54:12.742989 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:12.742954    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cc7a7a6-2c8d-426e-840c-3ef78e44e86a-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7nllb\" (UID: \"4cc7a7a6-2c8d-426e-840c-3ef78e44e86a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7nllb"
Nov 03 17:54:12.743472 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:12.743006    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nqn6\" (UniqueName: \"kubernetes.io/projected/4cc7a7a6-2c8d-426e-840c-3ef78e44e86a-kube-api-access-5nqn6\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7nllb\" (UID: \"4cc7a7a6-2c8d-426e-840c-3ef78e44e86a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7nllb"
Nov 03 17:54:12.743472 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:12.743036    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cc7a7a6-2c8d-426e-840c-3ef78e44e86a-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7nllb\" (UID: \"4cc7a7a6-2c8d-426e-840c-3ef78e44e86a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7nllb"
Nov 03 17:54:12.743472 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:12.743052    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cc7a7a6-2c8d-426e-840c-3ef78e44e86a-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7nllb\" (UID: \"4cc7a7a6-2c8d-426e-840c-3ef78e44e86a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7nllb"
Nov 03 17:54:12.743472 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:12.743081    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cc7a7a6-2c8d-426e-840c-3ef78e44e86a-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7nllb\" (UID: \"4cc7a7a6-2c8d-426e-840c-3ef78e44e86a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7nllb"
Nov 03 17:54:12.743472 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:12.743250    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cc7a7a6-2c8d-426e-840c-3ef78e44e86a-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7nllb\" (UID: \"4cc7a7a6-2c8d-426e-840c-3ef78e44e86a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7nllb"
Nov 03 17:54:12.743729 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:12.743520    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cc7a7a6-2c8d-426e-840c-3ef78e44e86a-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7nllb\" (UID: \"4cc7a7a6-2c8d-426e-840c-3ef78e44e86a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7nllb"
Nov 03 17:54:12.743729 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:12.743555    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cc7a7a6-2c8d-426e-840c-3ef78e44e86a-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7nllb\" (UID: \"4cc7a7a6-2c8d-426e-840c-3ef78e44e86a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7nllb"
Nov 03 17:54:12.743729 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:12.743581    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cc7a7a6-2c8d-426e-840c-3ef78e44e86a-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7nllb\" (UID: \"4cc7a7a6-2c8d-426e-840c-3ef78e44e86a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7nllb"
Nov 03 17:54:12.745294 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:12.745272    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cc7a7a6-2c8d-426e-840c-3ef78e44e86a-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7nllb\" (UID: \"4cc7a7a6-2c8d-426e-840c-3ef78e44e86a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7nllb"
Nov 03 17:54:12.745550 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:12.745530    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cc7a7a6-2c8d-426e-840c-3ef78e44e86a-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7nllb\" (UID: \"4cc7a7a6-2c8d-426e-840c-3ef78e44e86a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7nllb"
Nov 03 17:54:12.800259 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:12.800233    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nqn6\" (UniqueName: \"kubernetes.io/projected/4cc7a7a6-2c8d-426e-840c-3ef78e44e86a-kube-api-access-5nqn6\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7nllb\" (UID: \"4cc7a7a6-2c8d-426e-840c-3ef78e44e86a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7nllb"
Nov 03 17:54:12.840020 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:12.839987    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7nllb"
Nov 03 17:54:13.212695 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:13.212598    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-7nllb"]
Nov 03 17:54:13.220205 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:54:13.220176    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cc7a7a6_2c8d_426e_840c_3ef78e44e86a.slice/crio-43c46af07907f9e48e88bc75a54e33b8b6331faf4a3c2b738bac591f2692c8cb WatchSource:0}: Error finding container 43c46af07907f9e48e88bc75a54e33b8b6331faf4a3c2b738bac591f2692c8cb: Status 404 returned error can't find the container with id 43c46af07907f9e48e88bc75a54e33b8b6331faf4a3c2b738bac591f2692c8cb
Nov 03 17:54:13.840271 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:13.840234    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7nllb" event={"ID":"4cc7a7a6-2c8d-426e-840c-3ef78e44e86a","Type":"ContainerStarted","Data":"43c46af07907f9e48e88bc75a54e33b8b6331faf4a3c2b738bac591f2692c8cb"}
Nov 03 17:54:14.845245 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:14.845208    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7nllb" event={"ID":"4cc7a7a6-2c8d-426e-840c-3ef78e44e86a","Type":"ContainerStarted","Data":"4d9c17943b87649112eedc55f59da7acdb15ba6a7e413fca969d2e7df9232f81"}
Nov 03 17:54:14.955320 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:14.955260    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7nllb" podStartSLOduration=2.324585568 podStartE2EDuration="2.955247336s" totalImagesPullingTime="630.661768ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:54:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:54:14.954939853 +0000 UTC m=+493.971137919" watchObservedRunningTime="2026-11-03 17:54:14.955247336 +0000 UTC m=+493.971445404"
Nov 03 17:54:15.840580 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:15.840523    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7nllb"
Nov 03 17:54:18.755878 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:18.755845    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-d6rcp"]
Nov 03 17:54:18.759373 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:18.759349    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-d6rcp"
Nov 03 17:54:18.853784 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:18.853749    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Nov 03 17:54:18.854677 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:18.854658    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-d6rcp"]
Nov 03 17:54:18.895551 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:18.895517    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgj7l\" (UniqueName: \"kubernetes.io/projected/ee0c4b5d-37dd-482b-b5e9-4256a332a556-kube-api-access-qgj7l\") pod \"echo-simulator-kserve-64b6f8f96c-d6rcp\" (UID: \"ee0c4b5d-37dd-482b-b5e9-4256a332a556\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d6rcp"
Nov 03 17:54:18.895551 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:18.895556    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee0c4b5d-37dd-482b-b5e9-4256a332a556-home\") pod \"echo-simulator-kserve-64b6f8f96c-d6rcp\" (UID: \"ee0c4b5d-37dd-482b-b5e9-4256a332a556\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d6rcp"
Nov 03 17:54:18.895770 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:18.895591    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee0c4b5d-37dd-482b-b5e9-4256a332a556-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-d6rcp\" (UID: \"ee0c4b5d-37dd-482b-b5e9-4256a332a556\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d6rcp"
Nov 03 17:54:18.895770 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:18.895745    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee0c4b5d-37dd-482b-b5e9-4256a332a556-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-d6rcp\" (UID: \"ee0c4b5d-37dd-482b-b5e9-4256a332a556\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d6rcp"
Nov 03 17:54:18.895845 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:18.895776    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee0c4b5d-37dd-482b-b5e9-4256a332a556-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-d6rcp\" (UID: \"ee0c4b5d-37dd-482b-b5e9-4256a332a556\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d6rcp"
Nov 03 17:54:18.895845 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:18.895802    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee0c4b5d-37dd-482b-b5e9-4256a332a556-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-d6rcp\" (UID: \"ee0c4b5d-37dd-482b-b5e9-4256a332a556\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d6rcp"
Nov 03 17:54:18.997116 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:18.997083    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee0c4b5d-37dd-482b-b5e9-4256a332a556-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-d6rcp\" (UID: \"ee0c4b5d-37dd-482b-b5e9-4256a332a556\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d6rcp"
Nov 03 17:54:18.997287 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:18.997124    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee0c4b5d-37dd-482b-b5e9-4256a332a556-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-d6rcp\" (UID: \"ee0c4b5d-37dd-482b-b5e9-4256a332a556\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d6rcp"
Nov 03 17:54:18.997287 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:18.997144    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee0c4b5d-37dd-482b-b5e9-4256a332a556-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-d6rcp\" (UID: \"ee0c4b5d-37dd-482b-b5e9-4256a332a556\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d6rcp"
Nov 03 17:54:18.997287 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:18.997162    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee0c4b5d-37dd-482b-b5e9-4256a332a556-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-d6rcp\" (UID: \"ee0c4b5d-37dd-482b-b5e9-4256a332a556\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d6rcp"
Nov 03 17:54:18.997287 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:18.997212    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgj7l\" (UniqueName: \"kubernetes.io/projected/ee0c4b5d-37dd-482b-b5e9-4256a332a556-kube-api-access-qgj7l\") pod \"echo-simulator-kserve-64b6f8f96c-d6rcp\" (UID: \"ee0c4b5d-37dd-482b-b5e9-4256a332a556\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d6rcp"
Nov 03 17:54:18.997287 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:18.997252    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee0c4b5d-37dd-482b-b5e9-4256a332a556-home\") pod \"echo-simulator-kserve-64b6f8f96c-d6rcp\" (UID: \"ee0c4b5d-37dd-482b-b5e9-4256a332a556\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d6rcp"
Nov 03 17:54:18.997593 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:18.997564    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee0c4b5d-37dd-482b-b5e9-4256a332a556-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-d6rcp\" (UID: \"ee0c4b5d-37dd-482b-b5e9-4256a332a556\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d6rcp"
Nov 03 17:54:18.997766 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:18.997640    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee0c4b5d-37dd-482b-b5e9-4256a332a556-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-d6rcp\" (UID: \"ee0c4b5d-37dd-482b-b5e9-4256a332a556\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d6rcp"
Nov 03 17:54:18.997766 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:18.997711    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee0c4b5d-37dd-482b-b5e9-4256a332a556-home\") pod \"echo-simulator-kserve-64b6f8f96c-d6rcp\" (UID: \"ee0c4b5d-37dd-482b-b5e9-4256a332a556\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d6rcp"
Nov 03 17:54:19.000922 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:18.999797    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee0c4b5d-37dd-482b-b5e9-4256a332a556-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-d6rcp\" (UID: \"ee0c4b5d-37dd-482b-b5e9-4256a332a556\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d6rcp"
Nov 03 17:54:19.000922 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:19.000221    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee0c4b5d-37dd-482b-b5e9-4256a332a556-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-d6rcp\" (UID: \"ee0c4b5d-37dd-482b-b5e9-4256a332a556\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d6rcp"
Nov 03 17:54:19.055017 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:19.054984    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgj7l\" (UniqueName: \"kubernetes.io/projected/ee0c4b5d-37dd-482b-b5e9-4256a332a556-kube-api-access-qgj7l\") pod \"echo-simulator-kserve-64b6f8f96c-d6rcp\" (UID: \"ee0c4b5d-37dd-482b-b5e9-4256a332a556\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d6rcp"
Nov 03 17:54:19.071844 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:19.071816    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-d6rcp"
Nov 03 17:54:19.439237 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:19.439159    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-d6rcp"]
Nov 03 17:54:19.447212 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:54:19.447171    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee0c4b5d_37dd_482b_b5e9_4256a332a556.slice/crio-c21b4e235926f36edfd46f0b9fb388c3de36e67c5a51751cb8dd765d355edb93 WatchSource:0}: Error finding container c21b4e235926f36edfd46f0b9fb388c3de36e67c5a51751cb8dd765d355edb93: Status 404 returned error can't find the container with id c21b4e235926f36edfd46f0b9fb388c3de36e67c5a51751cb8dd765d355edb93
Nov 03 17:54:19.864047 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:19.864010    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-d6rcp" event={"ID":"ee0c4b5d-37dd-482b-b5e9-4256a332a556","Type":"ContainerStarted","Data":"c21b4e235926f36edfd46f0b9fb388c3de36e67c5a51751cb8dd765d355edb93"}
Nov 03 17:54:22.840998 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:22.840961    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7nllb"
Nov 03 17:54:22.853994 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:22.853962    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7nllb"
Nov 03 17:54:22.882378 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:22.882330    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-d6rcp" event={"ID":"ee0c4b5d-37dd-482b-b5e9-4256a332a556","Type":"ContainerStarted","Data":"bd5f63ab3de81d15db298dee0e07c92892906e8a6c17b20d2448717c1dfa109a"}
Nov 03 17:54:22.893372 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:22.893339    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7nllb"
Nov 03 17:54:23.072414 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:23.072383    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-d6rcp"
Nov 03 17:54:23.185990 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:23.185887    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-d6rcp" podStartSLOduration=2.517015608 podStartE2EDuration="5.185869595s" totalImagesPullingTime="2.668853987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:54:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:54:23.185310718 +0000 UTC m=+502.201508787" watchObservedRunningTime="2026-11-03 17:54:23.185869595 +0000 UTC m=+502.202067665"
Nov 03 17:54:24.576239 ip-10-0-128-204 kubenswrapper[5315]: E1103 17:54:24.576202    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 17:54:24.729883 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:24.729848    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7"]
Nov 03 17:54:24.733020 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:24.733000    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7"
Nov 03 17:54:24.827798 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:24.827718    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Nov 03 17:54:24.828632 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:24.828598    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7"]
Nov 03 17:54:24.847200 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:24.847171    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69760dba-faf9-4c08-895c-ad70fe95fd49-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7\" (UID: \"69760dba-faf9-4c08-895c-ad70fe95fd49\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7"
Nov 03 17:54:24.847424 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:24.847399    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69760dba-faf9-4c08-895c-ad70fe95fd49-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7\" (UID: \"69760dba-faf9-4c08-895c-ad70fe95fd49\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7"
Nov 03 17:54:24.847525 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:24.847436    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69760dba-faf9-4c08-895c-ad70fe95fd49-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7\" (UID: \"69760dba-faf9-4c08-895c-ad70fe95fd49\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7"
Nov 03 17:54:24.847525 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:24.847468    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69760dba-faf9-4c08-895c-ad70fe95fd49-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7\" (UID: \"69760dba-faf9-4c08-895c-ad70fe95fd49\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7"
Nov 03 17:54:24.847525 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:24.847503    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sxlb\" (UniqueName: \"kubernetes.io/projected/69760dba-faf9-4c08-895c-ad70fe95fd49-kube-api-access-7sxlb\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7\" (UID: \"69760dba-faf9-4c08-895c-ad70fe95fd49\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7"
Nov 03 17:54:24.847660 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:24.847564    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69760dba-faf9-4c08-895c-ad70fe95fd49-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7\" (UID: \"69760dba-faf9-4c08-895c-ad70fe95fd49\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7"
Nov 03 17:54:24.949003 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:24.948969    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69760dba-faf9-4c08-895c-ad70fe95fd49-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7\" (UID: \"69760dba-faf9-4c08-895c-ad70fe95fd49\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7"
Nov 03 17:54:24.949003 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:24.949005    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69760dba-faf9-4c08-895c-ad70fe95fd49-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7\" (UID: \"69760dba-faf9-4c08-895c-ad70fe95fd49\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7"
Nov 03 17:54:24.949207 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:24.949154    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sxlb\" (UniqueName: \"kubernetes.io/projected/69760dba-faf9-4c08-895c-ad70fe95fd49-kube-api-access-7sxlb\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7\" (UID: \"69760dba-faf9-4c08-895c-ad70fe95fd49\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7"
Nov 03 17:54:24.949207 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:24.949199    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69760dba-faf9-4c08-895c-ad70fe95fd49-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7\" (UID: \"69760dba-faf9-4c08-895c-ad70fe95fd49\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7"
Nov 03 17:54:24.949336 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:24.949314    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69760dba-faf9-4c08-895c-ad70fe95fd49-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7\" (UID: \"69760dba-faf9-4c08-895c-ad70fe95fd49\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7"
Nov 03 17:54:24.949541 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:24.949519    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69760dba-faf9-4c08-895c-ad70fe95fd49-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7\" (UID: \"69760dba-faf9-4c08-895c-ad70fe95fd49\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7"
Nov 03 17:54:24.949660 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:24.949428    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69760dba-faf9-4c08-895c-ad70fe95fd49-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7\" (UID: \"69760dba-faf9-4c08-895c-ad70fe95fd49\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7"
Nov 03 17:54:24.949660 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:24.949599    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69760dba-faf9-4c08-895c-ad70fe95fd49-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7\" (UID: \"69760dba-faf9-4c08-895c-ad70fe95fd49\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7"
Nov 03 17:54:24.949756 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:24.949710    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69760dba-faf9-4c08-895c-ad70fe95fd49-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7\" (UID: \"69760dba-faf9-4c08-895c-ad70fe95fd49\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7"
Nov 03 17:54:24.951387 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:24.951366    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69760dba-faf9-4c08-895c-ad70fe95fd49-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7\" (UID: \"69760dba-faf9-4c08-895c-ad70fe95fd49\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7"
Nov 03 17:54:24.951875 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:24.951858    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69760dba-faf9-4c08-895c-ad70fe95fd49-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7\" (UID: \"69760dba-faf9-4c08-895c-ad70fe95fd49\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7"
Nov 03 17:54:25.022022 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:25.021994    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sxlb\" (UniqueName: \"kubernetes.io/projected/69760dba-faf9-4c08-895c-ad70fe95fd49-kube-api-access-7sxlb\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7\" (UID: \"69760dba-faf9-4c08-895c-ad70fe95fd49\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7"
Nov 03 17:54:25.044087 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:25.044054    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7"
Nov 03 17:54:25.414842 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:25.414760    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7"]
Nov 03 17:54:25.421291 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:54:25.421259    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69760dba_faf9_4c08_895c_ad70fe95fd49.slice/crio-9eab4460136171eabf8dc09441b5e2bcf8d590eb4d73497a02a9decdbe86276f WatchSource:0}: Error finding container 9eab4460136171eabf8dc09441b5e2bcf8d590eb4d73497a02a9decdbe86276f: Status 404 returned error can't find the container with id 9eab4460136171eabf8dc09441b5e2bcf8d590eb4d73497a02a9decdbe86276f
Nov 03 17:54:25.894866 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:25.894827    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7" event={"ID":"69760dba-faf9-4c08-895c-ad70fe95fd49","Type":"ContainerStarted","Data":"9eab4460136171eabf8dc09441b5e2bcf8d590eb4d73497a02a9decdbe86276f"}
Nov 03 17:54:26.900454 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:26.900417    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7" event={"ID":"69760dba-faf9-4c08-895c-ad70fe95fd49","Type":"ContainerStarted","Data":"f304f6a70ae987e1100b7a3790a1cee52105f25805182675790120e4e8d8d6af"}
Nov 03 17:54:27.014398 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:27.014344    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7" podStartSLOduration=2.35976548 podStartE2EDuration="3.014331197s" totalImagesPullingTime="654.565717ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:54:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:54:27.013552559 +0000 UTC m=+506.029750630" watchObservedRunningTime="2026-11-03 17:54:27.014331197 +0000 UTC m=+506.030529265"
Nov 03 17:54:27.045187 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:27.045151    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7"
Nov 03 17:54:29.073022 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:29.072982    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-d6rcp"
Nov 03 17:54:29.085802 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:29.085769    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-d6rcp"
Nov 03 17:54:29.925402 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:29.925369    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-d6rcp"
Nov 03 17:54:35.044927 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:35.044888    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7"
Nov 03 17:54:35.057087 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:35.057055    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7"
Nov 03 17:54:35.947056 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:35.947028    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7"
Nov 03 17:54:54.576142 ip-10-0-128-204 kubenswrapper[5315]: E1103 17:54:54.576105    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 17:54:56.103288 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:56.103250    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c89f6d897-7c2rb"]
Nov 03 17:54:56.105370 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:56.105354    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c89f6d897-7c2rb"
Nov 03 17:54:56.118166 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:56.118144    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d8dacaea-58c6-4d11-ac3c-b7e234005a88-tls-cert\") pod \"authorino-c89f6d897-7c2rb\" (UID: \"d8dacaea-58c6-4d11-ac3c-b7e234005a88\") " pod="kuadrant-system/authorino-c89f6d897-7c2rb"
Nov 03 17:54:56.118252 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:56.118194    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj8rl\" (UniqueName: \"kubernetes.io/projected/d8dacaea-58c6-4d11-ac3c-b7e234005a88-kube-api-access-sj8rl\") pod \"authorino-c89f6d897-7c2rb\" (UID: \"d8dacaea-58c6-4d11-ac3c-b7e234005a88\") " pod="kuadrant-system/authorino-c89f6d897-7c2rb"
Nov 03 17:54:56.168452 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:56.168415    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c89f6d897-7c2rb"]
Nov 03 17:54:56.218860 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:56.218823    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d8dacaea-58c6-4d11-ac3c-b7e234005a88-tls-cert\") pod \"authorino-c89f6d897-7c2rb\" (UID: \"d8dacaea-58c6-4d11-ac3c-b7e234005a88\") " pod="kuadrant-system/authorino-c89f6d897-7c2rb"
Nov 03 17:54:56.219041 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:56.218890    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj8rl\" (UniqueName: \"kubernetes.io/projected/d8dacaea-58c6-4d11-ac3c-b7e234005a88-kube-api-access-sj8rl\") pod \"authorino-c89f6d897-7c2rb\" (UID: \"d8dacaea-58c6-4d11-ac3c-b7e234005a88\") " pod="kuadrant-system/authorino-c89f6d897-7c2rb"
Nov 03 17:54:56.221412 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:56.221385    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d8dacaea-58c6-4d11-ac3c-b7e234005a88-tls-cert\") pod \"authorino-c89f6d897-7c2rb\" (UID: \"d8dacaea-58c6-4d11-ac3c-b7e234005a88\") " pod="kuadrant-system/authorino-c89f6d897-7c2rb"
Nov 03 17:54:56.281085 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:56.281055    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj8rl\" (UniqueName: \"kubernetes.io/projected/d8dacaea-58c6-4d11-ac3c-b7e234005a88-kube-api-access-sj8rl\") pod \"authorino-c89f6d897-7c2rb\" (UID: \"d8dacaea-58c6-4d11-ac3c-b7e234005a88\") " pod="kuadrant-system/authorino-c89f6d897-7c2rb"
Nov 03 17:54:56.419185 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:56.419076    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c89f6d897-7c2rb"
Nov 03 17:54:56.791742 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:56.791709    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c89f6d897-7c2rb"]
Nov 03 17:54:56.800099 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:54:56.800064    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8dacaea_58c6_4d11_ac3c_b7e234005a88.slice/crio-5be0656badb7321bfd75e5d2c89d89929e4785ea1b8cff2342f911bbc74fabb4 WatchSource:0}: Error finding container 5be0656badb7321bfd75e5d2c89d89929e4785ea1b8cff2342f911bbc74fabb4: Status 404 returned error can't find the container with id 5be0656badb7321bfd75e5d2c89d89929e4785ea1b8cff2342f911bbc74fabb4
Nov 03 17:54:57.024904 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:54:57.024863    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c89f6d897-7c2rb" event={"ID":"d8dacaea-58c6-4d11-ac3c-b7e234005a88","Type":"ContainerStarted","Data":"5be0656badb7321bfd75e5d2c89d89929e4785ea1b8cff2342f911bbc74fabb4"}
Nov 03 17:55:04.061809 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:04.061779    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg"]
Nov 03 17:55:04.065263 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:04.065242    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg"
Nov 03 17:55:04.070815 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:04.070786    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwpld\" (UniqueName: \"kubernetes.io/projected/cb937207-ced4-4180-a554-0b7a86edce58-kube-api-access-gwpld\") pod \"maas-default-gateway-openshift-default-6dcf757899-g99pg\" (UID: \"cb937207-ced4-4180-a554-0b7a86edce58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg"
Nov 03 17:55:04.070955 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:04.070862    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb937207-ced4-4180-a554-0b7a86edce58-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-g99pg\" (UID: \"cb937207-ced4-4180-a554-0b7a86edce58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg"
Nov 03 17:55:04.070955 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:04.070888    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb937207-ced4-4180-a554-0b7a86edce58-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-g99pg\" (UID: \"cb937207-ced4-4180-a554-0b7a86edce58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg"
Nov 03 17:55:04.071074 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:04.070988    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb937207-ced4-4180-a554-0b7a86edce58-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-g99pg\" (UID: \"cb937207-ced4-4180-a554-0b7a86edce58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg"
Nov 03 17:55:04.071074 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:04.071034    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb937207-ced4-4180-a554-0b7a86edce58-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-g99pg\" (UID: \"cb937207-ced4-4180-a554-0b7a86edce58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg"
Nov 03 17:55:04.071074 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:04.071065    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb937207-ced4-4180-a554-0b7a86edce58-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-g99pg\" (UID: \"cb937207-ced4-4180-a554-0b7a86edce58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg"
Nov 03 17:55:04.071233 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:04.071098    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb937207-ced4-4180-a554-0b7a86edce58-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-g99pg\" (UID: \"cb937207-ced4-4180-a554-0b7a86edce58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg"
Nov 03 17:55:04.071233 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:04.071200    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb937207-ced4-4180-a554-0b7a86edce58-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-g99pg\" (UID: \"cb937207-ced4-4180-a554-0b7a86edce58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg"
Nov 03 17:55:04.071336 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:04.071231    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb937207-ced4-4180-a554-0b7a86edce58-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-g99pg\" (UID: \"cb937207-ced4-4180-a554-0b7a86edce58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg"
Nov 03 17:55:04.114163 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:04.114123    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-h54vm\""
Nov 03 17:55:04.171676 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:04.171645    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwpld\" (UniqueName: \"kubernetes.io/projected/cb937207-ced4-4180-a554-0b7a86edce58-kube-api-access-gwpld\") pod \"maas-default-gateway-openshift-default-6dcf757899-g99pg\" (UID: \"cb937207-ced4-4180-a554-0b7a86edce58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg"
Nov 03 17:55:04.171829 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:04.171685    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb937207-ced4-4180-a554-0b7a86edce58-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-g99pg\" (UID: \"cb937207-ced4-4180-a554-0b7a86edce58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg"
Nov 03 17:55:04.171829 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:04.171704    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb937207-ced4-4180-a554-0b7a86edce58-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-g99pg\" (UID: \"cb937207-ced4-4180-a554-0b7a86edce58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg"
Nov 03 17:55:04.171829 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:04.171722    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb937207-ced4-4180-a554-0b7a86edce58-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-g99pg\" (UID: \"cb937207-ced4-4180-a554-0b7a86edce58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg"
Nov 03 17:55:04.171998 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:04.171834    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb937207-ced4-4180-a554-0b7a86edce58-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-g99pg\" (UID: \"cb937207-ced4-4180-a554-0b7a86edce58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg"
Nov 03 17:55:04.171998 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:04.171869    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb937207-ced4-4180-a554-0b7a86edce58-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-g99pg\" (UID: \"cb937207-ced4-4180-a554-0b7a86edce58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg"
Nov 03 17:55:04.171998 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:04.171895    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb937207-ced4-4180-a554-0b7a86edce58-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-g99pg\" (UID: \"cb937207-ced4-4180-a554-0b7a86edce58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg"
Nov 03 17:55:04.171998 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:04.171975    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb937207-ced4-4180-a554-0b7a86edce58-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-g99pg\" (UID: \"cb937207-ced4-4180-a554-0b7a86edce58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg"
Nov 03 17:55:04.172184 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:04.172007    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb937207-ced4-4180-a554-0b7a86edce58-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-g99pg\" (UID: \"cb937207-ced4-4180-a554-0b7a86edce58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg"
Nov 03 17:55:04.172184 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:04.172132    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb937207-ced4-4180-a554-0b7a86edce58-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-g99pg\" (UID: \"cb937207-ced4-4180-a554-0b7a86edce58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg"
Nov 03 17:55:04.172305 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:04.172278    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb937207-ced4-4180-a554-0b7a86edce58-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-g99pg\" (UID: \"cb937207-ced4-4180-a554-0b7a86edce58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg"
Nov 03 17:55:04.172363 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:04.172302    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb937207-ced4-4180-a554-0b7a86edce58-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-g99pg\" (UID: \"cb937207-ced4-4180-a554-0b7a86edce58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg"
Nov 03 17:55:04.172363 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:04.172330    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb937207-ced4-4180-a554-0b7a86edce58-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-g99pg\" (UID: \"cb937207-ced4-4180-a554-0b7a86edce58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg"
Nov 03 17:55:04.172577 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:04.172524    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb937207-ced4-4180-a554-0b7a86edce58-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-g99pg\" (UID: \"cb937207-ced4-4180-a554-0b7a86edce58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg"
Nov 03 17:55:04.174417 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:04.174395    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb937207-ced4-4180-a554-0b7a86edce58-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-g99pg\" (UID: \"cb937207-ced4-4180-a554-0b7a86edce58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg"
Nov 03 17:55:04.174684 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:04.174664    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb937207-ced4-4180-a554-0b7a86edce58-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-g99pg\" (UID: \"cb937207-ced4-4180-a554-0b7a86edce58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg"
Nov 03 17:55:04.177591 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:04.177565    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg"]
Nov 03 17:55:04.229008 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:04.228978    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwpld\" (UniqueName: \"kubernetes.io/projected/cb937207-ced4-4180-a554-0b7a86edce58-kube-api-access-gwpld\") pod \"maas-default-gateway-openshift-default-6dcf757899-g99pg\" (UID: \"cb937207-ced4-4180-a554-0b7a86edce58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg"
Nov 03 17:55:04.229156 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:04.229021    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb937207-ced4-4180-a554-0b7a86edce58-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-g99pg\" (UID: \"cb937207-ced4-4180-a554-0b7a86edce58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg"
Nov 03 17:55:04.376525 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:04.376432    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg"
Nov 03 17:55:04.795042 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:04.795011    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg"]
Nov 03 17:55:04.800912 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:55:04.800884    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb937207_ced4_4180_a554_0b7a86edce58.slice/crio-3799fcdbec47096a4d333434f2778d26e625d7578162bf7adeabb2086974eeb9 WatchSource:0}: Error finding container 3799fcdbec47096a4d333434f2778d26e625d7578162bf7adeabb2086974eeb9: Status 404 returned error can't find the container with id 3799fcdbec47096a4d333434f2778d26e625d7578162bf7adeabb2086974eeb9
Nov 03 17:55:05.057123 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:05.057034    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg" event={"ID":"cb937207-ced4-4180-a554-0b7a86edce58","Type":"ContainerStarted","Data":"3799fcdbec47096a4d333434f2778d26e625d7578162bf7adeabb2086974eeb9"}
Nov 03 17:55:08.749723 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:08.749686    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 17:55:08.750015 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:08.749779    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 17:55:08.750015 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:08.749824    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 17:55:09.082448 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:09.082404    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg" event={"ID":"cb937207-ced4-4180-a554-0b7a86edce58","Type":"ContainerStarted","Data":"71e81734417c837c91e7b529296f44d2f7708dd73f25b9578a33d845895f0ba9"}
Nov 03 17:55:09.084493 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:09.084459    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c89f6d897-7c2rb" event={"ID":"d8dacaea-58c6-4d11-ac3c-b7e234005a88","Type":"ContainerStarted","Data":"16fb50d582c562197666ceadf04351c2d982cb0b411dcd4b2af606a3feea030c"}
Nov 03 17:55:09.247900 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:09.247822    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg" podStartSLOduration=1.30217553 podStartE2EDuration="5.24780233s" totalImagesPullingTime="3.9456268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:55:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:55:09.246946656 +0000 UTC m=+548.263144727" watchObservedRunningTime="2026-11-03 17:55:09.24780233 +0000 UTC m=+548.264000397"
Nov 03 17:55:09.354910 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:09.354798    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-c89f6d897-7c2rb" podStartSLOduration=1.472591709 podStartE2EDuration="13.354783259s" totalImagesPullingTime="11.88219155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:54:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:55:09.354774653 +0000 UTC m=+548.370972724" watchObservedRunningTime="2026-11-03 17:55:09.354783259 +0000 UTC m=+548.370981329"
Nov 03 17:55:09.376928 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:09.376892    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg"
Nov 03 17:55:09.378260 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:09.378229    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-g99pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Nov 03 17:55:09.378385 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:09.378309    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg" podUID="cb937207-ced4-4180-a554-0b7a86edce58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Nov 03 17:55:09.381031 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:09.381002    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-865f494c86-pmhmr"]
Nov 03 17:55:09.381244 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:09.381218    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-865f494c86-pmhmr" podUID="f53fdcc8-09c2-4c65-ae5b-c32cc3397b01" containerName="authorino" containerID="cri-o://1a299ca3550824c634716c211736913904729a63cacdfe63e83d52114f32b108" gracePeriod=30
Nov 03 17:55:09.394729 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:09.394695    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-h6mdw"]
Nov 03 17:55:09.398382 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:09.398358    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-h6mdw"
Nov 03 17:55:09.414634 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:09.414571    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhvkh\" (UniqueName: \"kubernetes.io/projected/fcd62b22-7b43-4f28-9f15-9066bc11e836-kube-api-access-xhvkh\") pod \"payload-pre-processing-7b98bb678c-h6mdw\" (UID: \"fcd62b22-7b43-4f28-9f15-9066bc11e836\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-h6mdw"
Nov 03 17:55:09.414752 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:09.414716    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcd62b22-7b43-4f28-9f15-9066bc11e836-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-h6mdw\" (UID: \"fcd62b22-7b43-4f28-9f15-9066bc11e836\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-h6mdw"
Nov 03 17:55:09.448636 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:09.446961    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 03 17:55:09.448636 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:09.447109    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-6v2pj\""
Nov 03 17:55:09.461981 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:09.461951    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-h6mdw"]
Nov 03 17:55:09.517934 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:09.516291    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhvkh\" (UniqueName: \"kubernetes.io/projected/fcd62b22-7b43-4f28-9f15-9066bc11e836-kube-api-access-xhvkh\") pod \"payload-pre-processing-7b98bb678c-h6mdw\" (UID: \"fcd62b22-7b43-4f28-9f15-9066bc11e836\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-h6mdw"
Nov 03 17:55:09.517934 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:09.516407    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcd62b22-7b43-4f28-9f15-9066bc11e836-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-h6mdw\" (UID: \"fcd62b22-7b43-4f28-9f15-9066bc11e836\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-h6mdw"
Nov 03 17:55:09.517934 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:09.517411    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcd62b22-7b43-4f28-9f15-9066bc11e836-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-h6mdw\" (UID: \"fcd62b22-7b43-4f28-9f15-9066bc11e836\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-h6mdw"
Nov 03 17:55:09.575221 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:09.575191    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhvkh\" (UniqueName: \"kubernetes.io/projected/fcd62b22-7b43-4f28-9f15-9066bc11e836-kube-api-access-xhvkh\") pod \"payload-pre-processing-7b98bb678c-h6mdw\" (UID: \"fcd62b22-7b43-4f28-9f15-9066bc11e836\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-h6mdw"
Nov 03 17:55:09.686260 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:09.686232    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-865f494c86-pmhmr"
Nov 03 17:55:09.719294 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:09.719257    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f53fdcc8-09c2-4c65-ae5b-c32cc3397b01-kube-api-access-6x2tn\" (UniqueName: \"kubernetes.io/projected/f53fdcc8-09c2-4c65-ae5b-c32cc3397b01-kube-api-access-6x2tn\") pod \"f53fdcc8-09c2-4c65-ae5b-c32cc3397b01\" (UID: \"f53fdcc8-09c2-4c65-ae5b-c32cc3397b01\") "
Nov 03 17:55:09.719478 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:09.719310    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f53fdcc8-09c2-4c65-ae5b-c32cc3397b01-tls-cert\" (UniqueName: \"kubernetes.io/secret/f53fdcc8-09c2-4c65-ae5b-c32cc3397b01-tls-cert\") pod \"f53fdcc8-09c2-4c65-ae5b-c32cc3397b01\" (UID: \"f53fdcc8-09c2-4c65-ae5b-c32cc3397b01\") "
Nov 03 17:55:09.721510 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:09.721470    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f53fdcc8-09c2-4c65-ae5b-c32cc3397b01-kube-api-access-6x2tn" pod "f53fdcc8-09c2-4c65-ae5b-c32cc3397b01" (UID: "f53fdcc8-09c2-4c65-ae5b-c32cc3397b01"). InnerVolumeSpecName "kube-api-access-6x2tn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:55:09.761384 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:09.761349    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f53fdcc8-09c2-4c65-ae5b-c32cc3397b01-tls-cert" pod "f53fdcc8-09c2-4c65-ae5b-c32cc3397b01" (UID: "f53fdcc8-09c2-4c65-ae5b-c32cc3397b01"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:55:09.773159 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:09.772913    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-h6mdw"
Nov 03 17:55:09.821270 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:09.821230    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x2tn\" (UniqueName: \"kubernetes.io/projected/f53fdcc8-09c2-4c65-ae5b-c32cc3397b01-kube-api-access-6x2tn\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:55:09.821270 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:09.821267    5315 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f53fdcc8-09c2-4c65-ae5b-c32cc3397b01-tls-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:55:10.090981 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:10.090940    5315 generic.go:356] "Generic (PLEG): container finished" podID="f53fdcc8-09c2-4c65-ae5b-c32cc3397b01" containerID="1a299ca3550824c634716c211736913904729a63cacdfe63e83d52114f32b108" exitCode=0
Nov 03 17:55:10.091142 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:10.090990    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-865f494c86-pmhmr"
Nov 03 17:55:10.091142 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:10.091025    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-865f494c86-pmhmr" event={"ID":"f53fdcc8-09c2-4c65-ae5b-c32cc3397b01","Type":"ContainerDied","Data":"1a299ca3550824c634716c211736913904729a63cacdfe63e83d52114f32b108"}
Nov 03 17:55:10.091142 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:10.091081    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-865f494c86-pmhmr" event={"ID":"f53fdcc8-09c2-4c65-ae5b-c32cc3397b01","Type":"ContainerDied","Data":"8897372951f1b862688be51ba101c756720aa2fd94559bbececcca43fbf77773"}
Nov 03 17:55:10.091142 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:10.091100    5315 scope.go:122] "RemoveContainer" containerID="1a299ca3550824c634716c211736913904729a63cacdfe63e83d52114f32b108"
Nov 03 17:55:10.100570 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:10.100549    5315 scope.go:122] "RemoveContainer" containerID="1a299ca3550824c634716c211736913904729a63cacdfe63e83d52114f32b108"
Nov 03 17:55:10.100858 ip-10-0-128-204 kubenswrapper[5315]: E1103 17:55:10.100838    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a299ca3550824c634716c211736913904729a63cacdfe63e83d52114f32b108\": container with ID starting with 1a299ca3550824c634716c211736913904729a63cacdfe63e83d52114f32b108 not found: ID does not exist" containerID="1a299ca3550824c634716c211736913904729a63cacdfe63e83d52114f32b108"
Nov 03 17:55:10.100905 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:10.100868    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a299ca3550824c634716c211736913904729a63cacdfe63e83d52114f32b108"} err="failed to get container status \"1a299ca3550824c634716c211736913904729a63cacdfe63e83d52114f32b108\": rpc error: code = NotFound desc = could not find container \"1a299ca3550824c634716c211736913904729a63cacdfe63e83d52114f32b108\": container with ID starting with 1a299ca3550824c634716c211736913904729a63cacdfe63e83d52114f32b108 not found: ID does not exist"
Nov 03 17:55:10.163939 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:10.163898    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-h6mdw"]
Nov 03 17:55:10.171784 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:55:10.171756    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcd62b22_7b43_4f28_9f15_9066bc11e836.slice/crio-48ef4ceca5e432c0a5bddc2331d29fb80cb9b583073e88b4ca88ee307c14e6b5 WatchSource:0}: Error finding container 48ef4ceca5e432c0a5bddc2331d29fb80cb9b583073e88b4ca88ee307c14e6b5: Status 404 returned error can't find the container with id 48ef4ceca5e432c0a5bddc2331d29fb80cb9b583073e88b4ca88ee307c14e6b5
Nov 03 17:55:10.249514 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:10.249408    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-865f494c86-pmhmr"]
Nov 03 17:55:10.254953 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:10.254914    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-865f494c86-pmhmr"]
Nov 03 17:55:10.377744 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:10.377656    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-g99pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Nov 03 17:55:10.377888 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:10.377742    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg" podUID="cb937207-ced4-4180-a554-0b7a86edce58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Nov 03 17:55:11.103784 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:11.103745    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f53fdcc8-09c2-4c65-ae5b-c32cc3397b01" path="/var/lib/kubelet/pods/f53fdcc8-09c2-4c65-ae5b-c32cc3397b01/volumes"
Nov 03 17:55:11.104161 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:11.104100    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-h6mdw" event={"ID":"fcd62b22-7b43-4f28-9f15-9066bc11e836","Type":"ContainerStarted","Data":"48ef4ceca5e432c0a5bddc2331d29fb80cb9b583073e88b4ca88ee307c14e6b5"}
Nov 03 17:55:11.377072 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:11.376978    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-g99pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Nov 03 17:55:11.377072 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:11.377054    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg" podUID="cb937207-ced4-4180-a554-0b7a86edce58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Nov 03 17:55:12.377514 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:12.377418    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-g99pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Nov 03 17:55:12.377514 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:12.377478    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg" podUID="cb937207-ced4-4180-a554-0b7a86edce58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Nov 03 17:55:13.377439 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:13.377395    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-g99pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Nov 03 17:55:13.377736 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:13.377469    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg" podUID="cb937207-ced4-4180-a554-0b7a86edce58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Nov 03 17:55:14.119533 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:14.119500    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-h6mdw" event={"ID":"fcd62b22-7b43-4f28-9f15-9066bc11e836","Type":"ContainerStarted","Data":"329b3940268e2106afbef5b4c2f23cff3870d922d89c34c8d7f67f1d365e276e"}
Nov 03 17:55:14.119726 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:14.119572    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-h6mdw"
Nov 03 17:55:14.232485 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:14.232416    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-h6mdw" podStartSLOduration=1.647988762 podStartE2EDuration="5.232394831s" totalImagesPullingTime="3.584406069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:55:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:55:14.231029943 +0000 UTC m=+553.247228016" watchObservedRunningTime="2026-11-03 17:55:14.232394831 +0000 UTC m=+553.248592902"
Nov 03 17:55:14.377515 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:14.377437    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-g99pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Nov 03 17:55:14.377663 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:14.377518    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg" podUID="cb937207-ced4-4180-a554-0b7a86edce58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Nov 03 17:55:15.377339 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:15.377295    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-g99pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Nov 03 17:55:15.377779 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:15.377368    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg" podUID="cb937207-ced4-4180-a554-0b7a86edce58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Nov 03 17:55:16.377575 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:16.377538    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-g99pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Nov 03 17:55:16.377987 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:16.377589    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg" podUID="cb937207-ced4-4180-a554-0b7a86edce58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Nov 03 17:55:18.243921 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:18.243886    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-64dd499c74-krd84"]
Nov 03 17:55:18.244431 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:18.244363    5315 cpu_manager.go:414] "containerMap: removing container" podUID="f53fdcc8-09c2-4c65-ae5b-c32cc3397b01" containerName="authorino"
Nov 03 17:55:18.244431 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:18.244382    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f53fdcc8-09c2-4c65-ae5b-c32cc3397b01" containerName="authorino"
Nov 03 17:55:18.244553 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:18.244480    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="f53fdcc8-09c2-4c65-ae5b-c32cc3397b01" containerName="authorino"
Nov 03 17:55:18.247454 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:18.247424    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-64dd499c74-krd84"
Nov 03 17:55:18.296180 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:18.296150    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 03 17:55:18.296180 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:18.296153    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Nov 03 17:55:18.296413 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:18.296344    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 03 17:55:18.340412 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:18.340363    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/186863ea-9176-4021-a742-3d26626a88a1-metrics-certs\") pod \"ai-gateway-controller-64dd499c74-krd84\" (UID: \"186863ea-9176-4021-a742-3d26626a88a1\") " pod="opendatahub/ai-gateway-controller-64dd499c74-krd84"
Nov 03 17:55:18.340602 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:18.340563    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2vn8\" (UniqueName: \"kubernetes.io/projected/186863ea-9176-4021-a742-3d26626a88a1-kube-api-access-v2vn8\") pod \"ai-gateway-controller-64dd499c74-krd84\" (UID: \"186863ea-9176-4021-a742-3d26626a88a1\") " pod="opendatahub/ai-gateway-controller-64dd499c74-krd84"
Nov 03 17:55:18.342663 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:18.342594    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-7wlk2\""
Nov 03 17:55:18.342810 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:18.342788    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Nov 03 17:55:18.343623 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:18.343584    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-64dd499c74-krd84"]
Nov 03 17:55:18.418821 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:18.418780    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-g99pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:55:18.419009 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:18.418841    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg" podUID="cb937207-ced4-4180-a554-0b7a86edce58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:55:18.441729 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:18.441680    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/186863ea-9176-4021-a742-3d26626a88a1-metrics-certs\") pod \"ai-gateway-controller-64dd499c74-krd84\" (UID: \"186863ea-9176-4021-a742-3d26626a88a1\") " pod="opendatahub/ai-gateway-controller-64dd499c74-krd84"
Nov 03 17:55:18.441906 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:18.441813    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2vn8\" (UniqueName: \"kubernetes.io/projected/186863ea-9176-4021-a742-3d26626a88a1-kube-api-access-v2vn8\") pod \"ai-gateway-controller-64dd499c74-krd84\" (UID: \"186863ea-9176-4021-a742-3d26626a88a1\") " pod="opendatahub/ai-gateway-controller-64dd499c74-krd84"
Nov 03 17:55:18.444423 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:18.444394    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/186863ea-9176-4021-a742-3d26626a88a1-metrics-certs\") pod \"ai-gateway-controller-64dd499c74-krd84\" (UID: \"186863ea-9176-4021-a742-3d26626a88a1\") " pod="opendatahub/ai-gateway-controller-64dd499c74-krd84"
Nov 03 17:55:18.500058 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:18.499982    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2vn8\" (UniqueName: \"kubernetes.io/projected/186863ea-9176-4021-a742-3d26626a88a1-kube-api-access-v2vn8\") pod \"ai-gateway-controller-64dd499c74-krd84\" (UID: \"186863ea-9176-4021-a742-3d26626a88a1\") " pod="opendatahub/ai-gateway-controller-64dd499c74-krd84"
Nov 03 17:55:18.562413 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:18.562370    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-64dd499c74-krd84"
Nov 03 17:55:18.933579 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:18.933540    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-64dd499c74-krd84"]
Nov 03 17:55:18.945244 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:55:18.945206    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod186863ea_9176_4021_a742_3d26626a88a1.slice/crio-0deb0b69faf1ad5615e936f3f59ea3a22c3fe03a7e6bdf562a1b520bfcc154b0 WatchSource:0}: Error finding container 0deb0b69faf1ad5615e936f3f59ea3a22c3fe03a7e6bdf562a1b520bfcc154b0: Status 404 returned error can't find the container with id 0deb0b69faf1ad5615e936f3f59ea3a22c3fe03a7e6bdf562a1b520bfcc154b0
Nov 03 17:55:19.146062 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:19.145996    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-64dd499c74-krd84" event={"ID":"186863ea-9176-4021-a742-3d26626a88a1","Type":"ContainerStarted","Data":"0deb0b69faf1ad5615e936f3f59ea3a22c3fe03a7e6bdf562a1b520bfcc154b0"}
Nov 03 17:55:19.260520 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:19.260444    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg"
Nov 03 17:55:19.260986 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:19.260903    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg"
Nov 03 17:55:19.261722 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:19.261693    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg"
Nov 03 17:55:22.161430 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:22.161396    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-64dd499c74-krd84" event={"ID":"186863ea-9176-4021-a742-3d26626a88a1","Type":"ContainerStarted","Data":"0229495581f444516729acafc655eb3d6d74f2f3cdca48e33df1691ca5d7f8fa"}
Nov 03 17:55:22.161872 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:22.161509    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-64dd499c74-krd84"
Nov 03 17:55:22.271167 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:22.271121    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-64dd499c74-krd84" podStartSLOduration=1.158739668 podStartE2EDuration="4.271107385s" totalImagesPullingTime="3.112367717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:55:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:55:22.270311198 +0000 UTC m=+561.286509269" watchObservedRunningTime="2026-11-03 17:55:22.271107385 +0000 UTC m=+561.287305454"
Nov 03 17:55:24.576783 ip-10-0-128-204 kubenswrapper[5315]: E1103 17:55:24.576747    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 17:55:25.127858 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:25.127831    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-h6mdw"
Nov 03 17:55:33.169043 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:55:33.169010    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-64dd499c74-krd84"
Nov 03 17:55:54.576300 ip-10-0-128-204 kubenswrapper[5315]: E1103 17:55:54.576255    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 17:56:01.138297 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:01.138264    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-97p65_c29d200a-c917-46e5-9cf5-21f4d952e470/ovn-acl-logging/0.log"
Nov 03 17:56:01.139676 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:01.139515    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-97p65_c29d200a-c917-46e5-9cf5-21f4d952e470/ovn-acl-logging/0.log"
Nov 03 17:56:06.929149 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:06.929111    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn"]
Nov 03 17:56:06.935909 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:06.935882    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn"
Nov 03 17:56:06.949086 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:06.949059    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb78e2af-5c83-4d6d-a562-274b12209c20-credential-socket\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn\" (UID: \"cb78e2af-5c83-4d6d-a562-274b12209c20\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn"
Nov 03 17:56:06.949241 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:06.949110    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb78e2af-5c83-4d6d-a562-274b12209c20-istio-podinfo\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn\" (UID: \"cb78e2af-5c83-4d6d-a562-274b12209c20\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn"
Nov 03 17:56:06.949241 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:06.949203    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv4nn\" (UniqueName: \"kubernetes.io/projected/cb78e2af-5c83-4d6d-a562-274b12209c20-kube-api-access-tv4nn\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn\" (UID: \"cb78e2af-5c83-4d6d-a562-274b12209c20\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn"
Nov 03 17:56:06.949241 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:06.949235    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb78e2af-5c83-4d6d-a562-274b12209c20-workload-socket\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn\" (UID: \"cb78e2af-5c83-4d6d-a562-274b12209c20\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn"
Nov 03 17:56:06.949365 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:06.949258    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb78e2af-5c83-4d6d-a562-274b12209c20-istio-token\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn\" (UID: \"cb78e2af-5c83-4d6d-a562-274b12209c20\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn"
Nov 03 17:56:06.949365 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:06.949340    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb78e2af-5c83-4d6d-a562-274b12209c20-istio-data\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn\" (UID: \"cb78e2af-5c83-4d6d-a562-274b12209c20\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn"
Nov 03 17:56:06.949476 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:06.949383    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb78e2af-5c83-4d6d-a562-274b12209c20-istio-envoy\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn\" (UID: \"cb78e2af-5c83-4d6d-a562-274b12209c20\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn"
Nov 03 17:56:06.949476 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:06.949407    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb78e2af-5c83-4d6d-a562-274b12209c20-workload-certs\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn\" (UID: \"cb78e2af-5c83-4d6d-a562-274b12209c20\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn"
Nov 03 17:56:06.949476 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:06.949427    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb78e2af-5c83-4d6d-a562-274b12209c20-istiod-ca-cert\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn\" (UID: \"cb78e2af-5c83-4d6d-a562-274b12209c20\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn"
Nov 03 17:56:06.985208 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:06.985182    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-f79c02-openshift-default-dockercfg-qxdds\""
Nov 03 17:56:07.031875 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.031842    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn"]
Nov 03 17:56:07.050723 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.050690    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb78e2af-5c83-4d6d-a562-274b12209c20-credential-socket\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn\" (UID: \"cb78e2af-5c83-4d6d-a562-274b12209c20\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn"
Nov 03 17:56:07.050882 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.050731    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb78e2af-5c83-4d6d-a562-274b12209c20-istio-podinfo\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn\" (UID: \"cb78e2af-5c83-4d6d-a562-274b12209c20\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn"
Nov 03 17:56:07.050882 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.050863    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv4nn\" (UniqueName: \"kubernetes.io/projected/cb78e2af-5c83-4d6d-a562-274b12209c20-kube-api-access-tv4nn\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn\" (UID: \"cb78e2af-5c83-4d6d-a562-274b12209c20\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn"
Nov 03 17:56:07.050995 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.050912    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb78e2af-5c83-4d6d-a562-274b12209c20-workload-socket\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn\" (UID: \"cb78e2af-5c83-4d6d-a562-274b12209c20\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn"
Nov 03 17:56:07.050995 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.050940    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb78e2af-5c83-4d6d-a562-274b12209c20-istio-token\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn\" (UID: \"cb78e2af-5c83-4d6d-a562-274b12209c20\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn"
Nov 03 17:56:07.050995 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.050974    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb78e2af-5c83-4d6d-a562-274b12209c20-istio-data\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn\" (UID: \"cb78e2af-5c83-4d6d-a562-274b12209c20\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn"
Nov 03 17:56:07.051153 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.051026    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb78e2af-5c83-4d6d-a562-274b12209c20-istio-envoy\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn\" (UID: \"cb78e2af-5c83-4d6d-a562-274b12209c20\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn"
Nov 03 17:56:07.051153 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.051061    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb78e2af-5c83-4d6d-a562-274b12209c20-workload-certs\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn\" (UID: \"cb78e2af-5c83-4d6d-a562-274b12209c20\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn"
Nov 03 17:56:07.051153 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.051092    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb78e2af-5c83-4d6d-a562-274b12209c20-istiod-ca-cert\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn\" (UID: \"cb78e2af-5c83-4d6d-a562-274b12209c20\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn"
Nov 03 17:56:07.051296 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.051152    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb78e2af-5c83-4d6d-a562-274b12209c20-credential-socket\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn\" (UID: \"cb78e2af-5c83-4d6d-a562-274b12209c20\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn"
Nov 03 17:56:07.051357 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.051328    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb78e2af-5c83-4d6d-a562-274b12209c20-workload-socket\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn\" (UID: \"cb78e2af-5c83-4d6d-a562-274b12209c20\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn"
Nov 03 17:56:07.051452 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.051433    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb78e2af-5c83-4d6d-a562-274b12209c20-istio-data\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn\" (UID: \"cb78e2af-5c83-4d6d-a562-274b12209c20\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn"
Nov 03 17:56:07.051562 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.051541    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb78e2af-5c83-4d6d-a562-274b12209c20-workload-certs\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn\" (UID: \"cb78e2af-5c83-4d6d-a562-274b12209c20\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn"
Nov 03 17:56:07.051764 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.051747    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb78e2af-5c83-4d6d-a562-274b12209c20-istiod-ca-cert\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn\" (UID: \"cb78e2af-5c83-4d6d-a562-274b12209c20\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn"
Nov 03 17:56:07.053338 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.053312    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb78e2af-5c83-4d6d-a562-274b12209c20-istio-envoy\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn\" (UID: \"cb78e2af-5c83-4d6d-a562-274b12209c20\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn"
Nov 03 17:56:07.053338 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.053325    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb78e2af-5c83-4d6d-a562-274b12209c20-istio-podinfo\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn\" (UID: \"cb78e2af-5c83-4d6d-a562-274b12209c20\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn"
Nov 03 17:56:07.109229 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.109192    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb78e2af-5c83-4d6d-a562-274b12209c20-istio-token\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn\" (UID: \"cb78e2af-5c83-4d6d-a562-274b12209c20\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn"
Nov 03 17:56:07.109229 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.109210    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv4nn\" (UniqueName: \"kubernetes.io/projected/cb78e2af-5c83-4d6d-a562-274b12209c20-kube-api-access-tv4nn\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn\" (UID: \"cb78e2af-5c83-4d6d-a562-274b12209c20\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn"
Nov 03 17:56:07.235278 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.235200    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855"]
Nov 03 17:56:07.237761 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.237735    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855"
Nov 03 17:56:07.246922 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.246899    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn"
Nov 03 17:56:07.252860 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.252835    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd5c9fcf-e132-42e2-b031-c608cfd0b392-workload-certs\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855\" (UID: \"dd5c9fcf-e132-42e2-b031-c608cfd0b392\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855"
Nov 03 17:56:07.252975 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.252875    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd5c9fcf-e132-42e2-b031-c608cfd0b392-istio-podinfo\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855\" (UID: \"dd5c9fcf-e132-42e2-b031-c608cfd0b392\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855"
Nov 03 17:56:07.252975 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.252897    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd5c9fcf-e132-42e2-b031-c608cfd0b392-istio-data\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855\" (UID: \"dd5c9fcf-e132-42e2-b031-c608cfd0b392\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855"
Nov 03 17:56:07.252975 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.252914    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd5c9fcf-e132-42e2-b031-c608cfd0b392-workload-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855\" (UID: \"dd5c9fcf-e132-42e2-b031-c608cfd0b392\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855"
Nov 03 17:56:07.253146 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.252979    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd5c9fcf-e132-42e2-b031-c608cfd0b392-istio-token\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855\" (UID: \"dd5c9fcf-e132-42e2-b031-c608cfd0b392\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855"
Nov 03 17:56:07.253146 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.253069    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd5c9fcf-e132-42e2-b031-c608cfd0b392-credential-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855\" (UID: \"dd5c9fcf-e132-42e2-b031-c608cfd0b392\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855"
Nov 03 17:56:07.253244 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.253172    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd5c9fcf-e132-42e2-b031-c608cfd0b392-istiod-ca-cert\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855\" (UID: \"dd5c9fcf-e132-42e2-b031-c608cfd0b392\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855"
Nov 03 17:56:07.253244 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.253212    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x6c4\" (UniqueName: \"kubernetes.io/projected/dd5c9fcf-e132-42e2-b031-c608cfd0b392-kube-api-access-2x6c4\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855\" (UID: \"dd5c9fcf-e132-42e2-b031-c608cfd0b392\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855"
Nov 03 17:56:07.253392 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.253264    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd5c9fcf-e132-42e2-b031-c608cfd0b392-istio-envoy\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855\" (UID: \"dd5c9fcf-e132-42e2-b031-c608cfd0b392\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855"
Nov 03 17:56:07.304839 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.304807    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855"]
Nov 03 17:56:07.310444 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.310419    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-ef9633-openshift-default-dockercfg-49fqv\""
Nov 03 17:56:07.353874 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.353836    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd5c9fcf-e132-42e2-b031-c608cfd0b392-istio-podinfo\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855\" (UID: \"dd5c9fcf-e132-42e2-b031-c608cfd0b392\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855"
Nov 03 17:56:07.353874 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.353878    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd5c9fcf-e132-42e2-b031-c608cfd0b392-istio-data\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855\" (UID: \"dd5c9fcf-e132-42e2-b031-c608cfd0b392\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855"
Nov 03 17:56:07.354076 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.353898    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd5c9fcf-e132-42e2-b031-c608cfd0b392-workload-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855\" (UID: \"dd5c9fcf-e132-42e2-b031-c608cfd0b392\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855"
Nov 03 17:56:07.354076 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.353917    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd5c9fcf-e132-42e2-b031-c608cfd0b392-istio-token\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855\" (UID: \"dd5c9fcf-e132-42e2-b031-c608cfd0b392\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855"
Nov 03 17:56:07.354076 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.353958    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd5c9fcf-e132-42e2-b031-c608cfd0b392-credential-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855\" (UID: \"dd5c9fcf-e132-42e2-b031-c608cfd0b392\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855"
Nov 03 17:56:07.354076 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.353980    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd5c9fcf-e132-42e2-b031-c608cfd0b392-istiod-ca-cert\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855\" (UID: \"dd5c9fcf-e132-42e2-b031-c608cfd0b392\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855"
Nov 03 17:56:07.354076 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.354009    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x6c4\" (UniqueName: \"kubernetes.io/projected/dd5c9fcf-e132-42e2-b031-c608cfd0b392-kube-api-access-2x6c4\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855\" (UID: \"dd5c9fcf-e132-42e2-b031-c608cfd0b392\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855"
Nov 03 17:56:07.354076 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.354051    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd5c9fcf-e132-42e2-b031-c608cfd0b392-istio-envoy\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855\" (UID: \"dd5c9fcf-e132-42e2-b031-c608cfd0b392\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855"
Nov 03 17:56:07.354366 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.354120    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd5c9fcf-e132-42e2-b031-c608cfd0b392-workload-certs\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855\" (UID: \"dd5c9fcf-e132-42e2-b031-c608cfd0b392\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855"
Nov 03 17:56:07.354366 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.354319    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd5c9fcf-e132-42e2-b031-c608cfd0b392-istio-data\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855\" (UID: \"dd5c9fcf-e132-42e2-b031-c608cfd0b392\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855"
Nov 03 17:56:07.354366 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.354343    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd5c9fcf-e132-42e2-b031-c608cfd0b392-credential-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855\" (UID: \"dd5c9fcf-e132-42e2-b031-c608cfd0b392\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855"
Nov 03 17:56:07.354511 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.354391    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd5c9fcf-e132-42e2-b031-c608cfd0b392-workload-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855\" (UID: \"dd5c9fcf-e132-42e2-b031-c608cfd0b392\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855"
Nov 03 17:56:07.354511 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.354479    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd5c9fcf-e132-42e2-b031-c608cfd0b392-workload-certs\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855\" (UID: \"dd5c9fcf-e132-42e2-b031-c608cfd0b392\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855"
Nov 03 17:56:07.354998 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.354956    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd5c9fcf-e132-42e2-b031-c608cfd0b392-istiod-ca-cert\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855\" (UID: \"dd5c9fcf-e132-42e2-b031-c608cfd0b392\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855"
Nov 03 17:56:07.356599 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.356577    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd5c9fcf-e132-42e2-b031-c608cfd0b392-istio-envoy\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855\" (UID: \"dd5c9fcf-e132-42e2-b031-c608cfd0b392\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855"
Nov 03 17:56:07.356708 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.356692    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd5c9fcf-e132-42e2-b031-c608cfd0b392-istio-podinfo\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855\" (UID: \"dd5c9fcf-e132-42e2-b031-c608cfd0b392\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855"
Nov 03 17:56:07.416747 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.416718    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x6c4\" (UniqueName: \"kubernetes.io/projected/dd5c9fcf-e132-42e2-b031-c608cfd0b392-kube-api-access-2x6c4\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855\" (UID: \"dd5c9fcf-e132-42e2-b031-c608cfd0b392\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855"
Nov 03 17:56:07.416889 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.416761    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd5c9fcf-e132-42e2-b031-c608cfd0b392-istio-token\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855\" (UID: \"dd5c9fcf-e132-42e2-b031-c608cfd0b392\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855"
Nov 03 17:56:07.549655 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.549600    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855"
Nov 03 17:56:07.673498 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.673463    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn"]
Nov 03 17:56:07.680944 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:56:07.680915    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb78e2af_5c83_4d6d_a562_274b12209c20.slice/crio-ff72f586bdc097b0d351cb419dfab78ceff515a5b33566e1951f64dedb078c1d WatchSource:0}: Error finding container ff72f586bdc097b0d351cb419dfab78ceff515a5b33566e1951f64dedb078c1d: Status 404 returned error can't find the container with id ff72f586bdc097b0d351cb419dfab78ceff515a5b33566e1951f64dedb078c1d
Nov 03 17:56:07.683034 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.683000    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 17:56:07.683123 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.683067    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 17:56:07.683123 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.683094    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 17:56:07.953468 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.953389    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855"]
Nov 03 17:56:07.961650 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:56:07.961603    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd5c9fcf_e132_42e2_b031_c608cfd0b392.slice/crio-e3d6c6c99680f0561fed9780f40861c6e8802a42d3845b64572c85e476fcc88b WatchSource:0}: Error finding container e3d6c6c99680f0561fed9780f40861c6e8802a42d3845b64572c85e476fcc88b: Status 404 returned error can't find the container with id e3d6c6c99680f0561fed9780f40861c6e8802a42d3845b64572c85e476fcc88b
Nov 03 17:56:07.964445 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.964406    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 17:56:07.964550 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.964483    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 17:56:07.964550 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:07.964513    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 17:56:08.358683 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:08.358601    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn" event={"ID":"cb78e2af-5c83-4d6d-a562-274b12209c20","Type":"ContainerStarted","Data":"ec2b17e223560519400e4f7691fa64d88d00bbd8d54624bfb48290322d244893"}
Nov 03 17:56:08.358856 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:08.358700    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn" event={"ID":"cb78e2af-5c83-4d6d-a562-274b12209c20","Type":"ContainerStarted","Data":"ff72f586bdc097b0d351cb419dfab78ceff515a5b33566e1951f64dedb078c1d"}
Nov 03 17:56:08.360563 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:08.360533    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855" event={"ID":"dd5c9fcf-e132-42e2-b031-c608cfd0b392","Type":"ContainerStarted","Data":"f2a6ea1f91953839e03ec6653e0e9e5ef8b0e68ec3835cf1c567f56ff182207a"}
Nov 03 17:56:08.360767 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:08.360586    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855" event={"ID":"dd5c9fcf-e132-42e2-b031-c608cfd0b392","Type":"ContainerStarted","Data":"e3d6c6c99680f0561fed9780f40861c6e8802a42d3845b64572c85e476fcc88b"}
Nov 03 17:56:08.501430 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:08.501377    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn" podStartSLOduration=2.501363702 podStartE2EDuration="2.501363702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:08.500260687 +0000 UTC m=+607.516458755" watchObservedRunningTime="2026-11-03 17:56:08.501363702 +0000 UTC m=+607.517561764"
Nov 03 17:56:08.549720 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:08.549689    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855"
Nov 03 17:56:08.659730 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:08.659585    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855" podStartSLOduration=1.659565736 podStartE2EDuration="1.659565736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:08.659562982 +0000 UTC m=+607.675761050" watchObservedRunningTime="2026-11-03 17:56:08.659565736 +0000 UTC m=+607.675763806"
Nov 03 17:56:08.662292 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:08.662266    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9"]
Nov 03 17:56:08.664953 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:08.664932    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9"
Nov 03 17:56:08.721535 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:08.721505    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-0adcbc-openshift-default-dockercfg-zctsw\""
Nov 03 17:56:08.729903 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:08.729877    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9"]
Nov 03 17:56:08.767237 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:08.767203    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25a8498d-e23e-44a9-81fc-3170b124ad13-workload-certs\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9\" (UID: \"25a8498d-e23e-44a9-81fc-3170b124ad13\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9"
Nov 03 17:56:08.767237 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:08.767237    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25a8498d-e23e-44a9-81fc-3170b124ad13-istio-podinfo\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9\" (UID: \"25a8498d-e23e-44a9-81fc-3170b124ad13\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9"
Nov 03 17:56:08.767443 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:08.767256    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25a8498d-e23e-44a9-81fc-3170b124ad13-istio-token\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9\" (UID: \"25a8498d-e23e-44a9-81fc-3170b124ad13\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9"
Nov 03 17:56:08.767443 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:08.767350    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a8498d-e23e-44a9-81fc-3170b124ad13-workload-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9\" (UID: \"25a8498d-e23e-44a9-81fc-3170b124ad13\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9"
Nov 03 17:56:08.767443 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:08.767392    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp2d5\" (UniqueName: \"kubernetes.io/projected/25a8498d-e23e-44a9-81fc-3170b124ad13-kube-api-access-rp2d5\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9\" (UID: \"25a8498d-e23e-44a9-81fc-3170b124ad13\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9"
Nov 03 17:56:08.767443 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:08.767428    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25a8498d-e23e-44a9-81fc-3170b124ad13-istiod-ca-cert\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9\" (UID: \"25a8498d-e23e-44a9-81fc-3170b124ad13\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9"
Nov 03 17:56:08.767567 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:08.767501    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25a8498d-e23e-44a9-81fc-3170b124ad13-istio-envoy\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9\" (UID: \"25a8498d-e23e-44a9-81fc-3170b124ad13\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9"
Nov 03 17:56:08.767567 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:08.767560    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a8498d-e23e-44a9-81fc-3170b124ad13-credential-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9\" (UID: \"25a8498d-e23e-44a9-81fc-3170b124ad13\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9"
Nov 03 17:56:08.767647 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:08.767639    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25a8498d-e23e-44a9-81fc-3170b124ad13-istio-data\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9\" (UID: \"25a8498d-e23e-44a9-81fc-3170b124ad13\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9"
Nov 03 17:56:08.868357 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:08.868322    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a8498d-e23e-44a9-81fc-3170b124ad13-workload-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9\" (UID: \"25a8498d-e23e-44a9-81fc-3170b124ad13\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9"
Nov 03 17:56:08.868357 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:08.868368    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp2d5\" (UniqueName: \"kubernetes.io/projected/25a8498d-e23e-44a9-81fc-3170b124ad13-kube-api-access-rp2d5\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9\" (UID: \"25a8498d-e23e-44a9-81fc-3170b124ad13\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9"
Nov 03 17:56:08.868602 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:08.868411    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25a8498d-e23e-44a9-81fc-3170b124ad13-istiod-ca-cert\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9\" (UID: \"25a8498d-e23e-44a9-81fc-3170b124ad13\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9"
Nov 03 17:56:08.868602 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:08.868461    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25a8498d-e23e-44a9-81fc-3170b124ad13-istio-envoy\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9\" (UID: \"25a8498d-e23e-44a9-81fc-3170b124ad13\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9"
Nov 03 17:56:08.868602 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:08.868514    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a8498d-e23e-44a9-81fc-3170b124ad13-credential-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9\" (UID: \"25a8498d-e23e-44a9-81fc-3170b124ad13\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9"
Nov 03 17:56:08.868602 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:08.868581    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25a8498d-e23e-44a9-81fc-3170b124ad13-istio-data\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9\" (UID: \"25a8498d-e23e-44a9-81fc-3170b124ad13\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9"
Nov 03 17:56:08.868840 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:08.868648    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25a8498d-e23e-44a9-81fc-3170b124ad13-workload-certs\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9\" (UID: \"25a8498d-e23e-44a9-81fc-3170b124ad13\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9"
Nov 03 17:56:08.868840 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:08.868699    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25a8498d-e23e-44a9-81fc-3170b124ad13-istio-podinfo\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9\" (UID: \"25a8498d-e23e-44a9-81fc-3170b124ad13\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9"
Nov 03 17:56:08.868840 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:08.868741    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25a8498d-e23e-44a9-81fc-3170b124ad13-istio-token\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9\" (UID: \"25a8498d-e23e-44a9-81fc-3170b124ad13\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9"
Nov 03 17:56:08.868840 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:08.868803    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a8498d-e23e-44a9-81fc-3170b124ad13-workload-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9\" (UID: \"25a8498d-e23e-44a9-81fc-3170b124ad13\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9"
Nov 03 17:56:08.869038 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:08.868933    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a8498d-e23e-44a9-81fc-3170b124ad13-credential-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9\" (UID: \"25a8498d-e23e-44a9-81fc-3170b124ad13\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9"
Nov 03 17:56:08.869093 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:08.869043    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25a8498d-e23e-44a9-81fc-3170b124ad13-istio-data\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9\" (UID: \"25a8498d-e23e-44a9-81fc-3170b124ad13\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9"
Nov 03 17:56:08.869156 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:08.869134    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25a8498d-e23e-44a9-81fc-3170b124ad13-workload-certs\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9\" (UID: \"25a8498d-e23e-44a9-81fc-3170b124ad13\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9"
Nov 03 17:56:08.869343 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:08.869317    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25a8498d-e23e-44a9-81fc-3170b124ad13-istiod-ca-cert\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9\" (UID: \"25a8498d-e23e-44a9-81fc-3170b124ad13\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9"
Nov 03 17:56:08.871042 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:08.871013    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25a8498d-e23e-44a9-81fc-3170b124ad13-istio-envoy\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9\" (UID: \"25a8498d-e23e-44a9-81fc-3170b124ad13\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9"
Nov 03 17:56:08.871374 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:08.871352    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25a8498d-e23e-44a9-81fc-3170b124ad13-istio-podinfo\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9\" (UID: \"25a8498d-e23e-44a9-81fc-3170b124ad13\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9"
Nov 03 17:56:08.929087 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:08.929021    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25a8498d-e23e-44a9-81fc-3170b124ad13-istio-token\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9\" (UID: \"25a8498d-e23e-44a9-81fc-3170b124ad13\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9"
Nov 03 17:56:08.931590 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:08.931562    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp2d5\" (UniqueName: \"kubernetes.io/projected/25a8498d-e23e-44a9-81fc-3170b124ad13-kube-api-access-rp2d5\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9\" (UID: \"25a8498d-e23e-44a9-81fc-3170b124ad13\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9"
Nov 03 17:56:08.952739 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:08.952701    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45"]
Nov 03 17:56:08.955754 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:08.955735    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45"
Nov 03 17:56:08.969825 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:08.969800    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e8997a5-bfb9-4517-bb5e-632f8639e44d-istio-podinfo\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45\" (UID: \"4e8997a5-bfb9-4517-bb5e-632f8639e44d\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45"
Nov 03 17:56:08.969950 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:08.969830    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e8997a5-bfb9-4517-bb5e-632f8639e44d-istiod-ca-cert\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45\" (UID: \"4e8997a5-bfb9-4517-bb5e-632f8639e44d\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45"
Nov 03 17:56:08.969950 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:08.969848    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e8997a5-bfb9-4517-bb5e-632f8639e44d-istio-envoy\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45\" (UID: \"4e8997a5-bfb9-4517-bb5e-632f8639e44d\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45"
Nov 03 17:56:08.969950 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:08.969876    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e8997a5-bfb9-4517-bb5e-632f8639e44d-istio-token\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45\" (UID: \"4e8997a5-bfb9-4517-bb5e-632f8639e44d\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45"
Nov 03 17:56:08.970122 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:08.970041    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8997a5-bfb9-4517-bb5e-632f8639e44d-credential-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45\" (UID: \"4e8997a5-bfb9-4517-bb5e-632f8639e44d\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45"
Nov 03 17:56:08.970122 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:08.970080    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e8997a5-bfb9-4517-bb5e-632f8639e44d-istio-data\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45\" (UID: \"4e8997a5-bfb9-4517-bb5e-632f8639e44d\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45"
Nov 03 17:56:08.970187 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:08.970150    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whghb\" (UniqueName: \"kubernetes.io/projected/4e8997a5-bfb9-4517-bb5e-632f8639e44d-kube-api-access-whghb\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45\" (UID: \"4e8997a5-bfb9-4517-bb5e-632f8639e44d\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45"
Nov 03 17:56:08.970223 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:08.970198    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e8997a5-bfb9-4517-bb5e-632f8639e44d-workload-certs\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45\" (UID: \"4e8997a5-bfb9-4517-bb5e-632f8639e44d\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45"
Nov 03 17:56:08.970299 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:08.970285    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8997a5-bfb9-4517-bb5e-632f8639e44d-workload-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45\" (UID: \"4e8997a5-bfb9-4517-bb5e-632f8639e44d\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45"
Nov 03 17:56:08.977762 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:08.977741    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9"
Nov 03 17:56:09.032098 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:09.031359    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-a86228-openshift-default-dockercfg-pqbnk\""
Nov 03 17:56:09.054720 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:09.054686    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45"]
Nov 03 17:56:09.070935 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:09.070904    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8997a5-bfb9-4517-bb5e-632f8639e44d-credential-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45\" (UID: \"4e8997a5-bfb9-4517-bb5e-632f8639e44d\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45"
Nov 03 17:56:09.070935 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:09.070936    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e8997a5-bfb9-4517-bb5e-632f8639e44d-istio-data\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45\" (UID: \"4e8997a5-bfb9-4517-bb5e-632f8639e44d\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45"
Nov 03 17:56:09.071149 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:09.070966    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whghb\" (UniqueName: \"kubernetes.io/projected/4e8997a5-bfb9-4517-bb5e-632f8639e44d-kube-api-access-whghb\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45\" (UID: \"4e8997a5-bfb9-4517-bb5e-632f8639e44d\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45"
Nov 03 17:56:09.071149 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:09.071002    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e8997a5-bfb9-4517-bb5e-632f8639e44d-workload-certs\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45\" (UID: \"4e8997a5-bfb9-4517-bb5e-632f8639e44d\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45"
Nov 03 17:56:09.071149 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:09.071033    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8997a5-bfb9-4517-bb5e-632f8639e44d-workload-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45\" (UID: \"4e8997a5-bfb9-4517-bb5e-632f8639e44d\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45"
Nov 03 17:56:09.071149 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:09.071054    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e8997a5-bfb9-4517-bb5e-632f8639e44d-istio-podinfo\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45\" (UID: \"4e8997a5-bfb9-4517-bb5e-632f8639e44d\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45"
Nov 03 17:56:09.071149 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:09.071069    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e8997a5-bfb9-4517-bb5e-632f8639e44d-istiod-ca-cert\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45\" (UID: \"4e8997a5-bfb9-4517-bb5e-632f8639e44d\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45"
Nov 03 17:56:09.071149 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:09.071087    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e8997a5-bfb9-4517-bb5e-632f8639e44d-istio-envoy\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45\" (UID: \"4e8997a5-bfb9-4517-bb5e-632f8639e44d\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45"
Nov 03 17:56:09.071149 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:09.071110    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e8997a5-bfb9-4517-bb5e-632f8639e44d-istio-token\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45\" (UID: \"4e8997a5-bfb9-4517-bb5e-632f8639e44d\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45"
Nov 03 17:56:09.071412 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:09.071387    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8997a5-bfb9-4517-bb5e-632f8639e44d-credential-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45\" (UID: \"4e8997a5-bfb9-4517-bb5e-632f8639e44d\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45"
Nov 03 17:56:09.071451 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:09.071405    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e8997a5-bfb9-4517-bb5e-632f8639e44d-istio-data\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45\" (UID: \"4e8997a5-bfb9-4517-bb5e-632f8639e44d\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45"
Nov 03 17:56:09.071783 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:09.071758    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e8997a5-bfb9-4517-bb5e-632f8639e44d-workload-certs\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45\" (UID: \"4e8997a5-bfb9-4517-bb5e-632f8639e44d\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45"
Nov 03 17:56:09.071920 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:09.071903    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8997a5-bfb9-4517-bb5e-632f8639e44d-workload-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45\" (UID: \"4e8997a5-bfb9-4517-bb5e-632f8639e44d\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45"
Nov 03 17:56:09.072441 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:09.072416    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e8997a5-bfb9-4517-bb5e-632f8639e44d-istiod-ca-cert\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45\" (UID: \"4e8997a5-bfb9-4517-bb5e-632f8639e44d\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45"
Nov 03 17:56:09.079639 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:09.074182    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e8997a5-bfb9-4517-bb5e-632f8639e44d-istio-envoy\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45\" (UID: \"4e8997a5-bfb9-4517-bb5e-632f8639e44d\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45"
Nov 03 17:56:09.079639 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:09.074503    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e8997a5-bfb9-4517-bb5e-632f8639e44d-istio-podinfo\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45\" (UID: \"4e8997a5-bfb9-4517-bb5e-632f8639e44d\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45"
Nov 03 17:56:09.140047 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:09.140010    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whghb\" (UniqueName: \"kubernetes.io/projected/4e8997a5-bfb9-4517-bb5e-632f8639e44d-kube-api-access-whghb\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45\" (UID: \"4e8997a5-bfb9-4517-bb5e-632f8639e44d\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45"
Nov 03 17:56:09.140186 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:09.140161    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e8997a5-bfb9-4517-bb5e-632f8639e44d-istio-token\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45\" (UID: \"4e8997a5-bfb9-4517-bb5e-632f8639e44d\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45"
Nov 03 17:56:09.248057 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:09.247983    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn"
Nov 03 17:56:09.253237 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:09.253211    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn"
Nov 03 17:56:09.267132 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:09.267107    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45"
Nov 03 17:56:09.364853 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:09.364822    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn"
Nov 03 17:56:09.365727 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:09.365704    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn"
Nov 03 17:56:09.403112 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:09.403084    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9"]
Nov 03 17:56:09.410815 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:56:09.410789    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25a8498d_e23e_44a9_81fc_3170b124ad13.slice/crio-e76c9ed54e2908688ec36d5ccf481f43b623ec6da50331a6fca414c696fb549e WatchSource:0}: Error finding container e76c9ed54e2908688ec36d5ccf481f43b623ec6da50331a6fca414c696fb549e: Status 404 returned error can't find the container with id e76c9ed54e2908688ec36d5ccf481f43b623ec6da50331a6fca414c696fb549e
Nov 03 17:56:09.412984 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:09.412955    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 17:56:09.413061 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:09.413020    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 17:56:09.413061 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:09.413051    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 17:56:09.556075 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:09.556046    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855"
Nov 03 17:56:09.646068 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:09.646035    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45"]
Nov 03 17:56:09.652588 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:56:09.652557    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e8997a5_bfb9_4517_bb5e_632f8639e44d.slice/crio-c06b0303795abc7bbb3d8c25a5cff37c3b88c67e5a48389732485dd3a4b68046 WatchSource:0}: Error finding container c06b0303795abc7bbb3d8c25a5cff37c3b88c67e5a48389732485dd3a4b68046: Status 404 returned error can't find the container with id c06b0303795abc7bbb3d8c25a5cff37c3b88c67e5a48389732485dd3a4b68046
Nov 03 17:56:09.655005 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:09.654971    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 17:56:09.655112 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:09.655037    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 17:56:09.655112 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:09.655067    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 17:56:10.372869 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:10.372832    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9" event={"ID":"25a8498d-e23e-44a9-81fc-3170b124ad13","Type":"ContainerStarted","Data":"0d23428114557db8e457dc9944c3de90e0c1ea9a9527b88683aca4ec1b9a423f"}
Nov 03 17:56:10.373290 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:10.372941    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9" event={"ID":"25a8498d-e23e-44a9-81fc-3170b124ad13","Type":"ContainerStarted","Data":"e76c9ed54e2908688ec36d5ccf481f43b623ec6da50331a6fca414c696fb549e"}
Nov 03 17:56:10.374443 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:10.374413    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45" event={"ID":"4e8997a5-bfb9-4517-bb5e-632f8639e44d","Type":"ContainerStarted","Data":"6afcb66d3e49bf9e99b751a2306aa56d55b92c13a671ac2d80588a967d564ce2"}
Nov 03 17:56:10.374557 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:10.374456    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45" event={"ID":"4e8997a5-bfb9-4517-bb5e-632f8639e44d","Type":"ContainerStarted","Data":"c06b0303795abc7bbb3d8c25a5cff37c3b88c67e5a48389732485dd3a4b68046"}
Nov 03 17:56:10.374657 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:10.374638    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855"
Nov 03 17:56:10.375656 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:10.375636    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855"
Nov 03 17:56:10.488183 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:10.488121    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9" podStartSLOduration=2.488101264 podStartE2EDuration="2.488101264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:10.485813974 +0000 UTC m=+609.502012043" watchObservedRunningTime="2026-11-03 17:56:10.488101264 +0000 UTC m=+609.504299335"
Nov 03 17:56:10.605743 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:10.605659    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45" podStartSLOduration=2.605638743 podStartE2EDuration="2.605638743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:10.604827628 +0000 UTC m=+609.621025698" watchObservedRunningTime="2026-11-03 17:56:10.605638743 +0000 UTC m=+609.621836813"
Nov 03 17:56:10.978163 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:10.978132    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9"
Nov 03 17:56:10.983262 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:10.983235    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9"
Nov 03 17:56:11.268259 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:11.268185    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45"
Nov 03 17:56:11.272955 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:11.272931    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45"
Nov 03 17:56:11.379544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:11.379513    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9"
Nov 03 17:56:11.379957 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:11.379562    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45"
Nov 03 17:56:11.380111 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:11.380094    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45"
Nov 03 17:56:11.380190 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:11.380160    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9"
Nov 03 17:56:16.930422 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:16.930387    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-f79c02-5477cd8fff-2h4nk"]
Nov 03 17:56:16.932598 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:16.932581    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f79c02-5477cd8fff-2h4nk"
Nov 03 17:56:17.037810 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:17.037777    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-f79c02-dockercfg-56rt5\""
Nov 03 17:56:17.037810 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:17.037776    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-f79c02\""
Nov 03 17:56:17.043565 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:17.043511    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxh6d\" (UniqueName: \"kubernetes.io/projected/13fcfa33-03ce-4384-a431-edf17a1f2735-kube-api-access-sxh6d\") pod \"payload-pre-processing-e2e-worker-w2-f79c02-5477cd8fff-2h4nk\" (UID: \"13fcfa33-03ce-4384-a431-edf17a1f2735\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f79c02-5477cd8fff-2h4nk"
Nov 03 17:56:17.043565 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:17.043568    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13fcfa33-03ce-4384-a431-edf17a1f2735-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-f79c02-5477cd8fff-2h4nk\" (UID: \"13fcfa33-03ce-4384-a431-edf17a1f2735\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f79c02-5477cd8fff-2h4nk"
Nov 03 17:56:17.071412 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:17.068442    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-f79c02-5477cd8fff-2h4nk"]
Nov 03 17:56:17.131805 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:17.131771    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-f79c02-5c7b8dcbf7-qtwl6"]
Nov 03 17:56:17.134288 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:17.134265    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-f79c02-5c7b8dcbf7-qtwl6"
Nov 03 17:56:17.144503 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:17.144452    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxh6d\" (UniqueName: \"kubernetes.io/projected/13fcfa33-03ce-4384-a431-edf17a1f2735-kube-api-access-sxh6d\") pod \"payload-pre-processing-e2e-worker-w2-f79c02-5477cd8fff-2h4nk\" (UID: \"13fcfa33-03ce-4384-a431-edf17a1f2735\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f79c02-5477cd8fff-2h4nk"
Nov 03 17:56:17.144764 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:17.144742    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13fcfa33-03ce-4384-a431-edf17a1f2735-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-f79c02-5477cd8fff-2h4nk\" (UID: \"13fcfa33-03ce-4384-a431-edf17a1f2735\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f79c02-5477cd8fff-2h4nk"
Nov 03 17:56:17.145677 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:17.145655    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13fcfa33-03ce-4384-a431-edf17a1f2735-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-f79c02-5477cd8fff-2h4nk\" (UID: \"13fcfa33-03ce-4384-a431-edf17a1f2735\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f79c02-5477cd8fff-2h4nk"
Nov 03 17:56:17.202077 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:17.201994    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-f79c02-5c7b8dcbf7-qtwl6"]
Nov 03 17:56:17.208352 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:17.208317    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxh6d\" (UniqueName: \"kubernetes.io/projected/13fcfa33-03ce-4384-a431-edf17a1f2735-kube-api-access-sxh6d\") pod \"payload-pre-processing-e2e-worker-w2-f79c02-5477cd8fff-2h4nk\" (UID: \"13fcfa33-03ce-4384-a431-edf17a1f2735\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f79c02-5477cd8fff-2h4nk"
Nov 03 17:56:17.245933 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:17.245896    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7861e9eb-a526-442b-800f-6009fa55789a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-f79c02-5c7b8dcbf7-qtwl6\" (UID: \"7861e9eb-a526-442b-800f-6009fa55789a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-f79c02-5c7b8dcbf7-qtwl6"
Nov 03 17:56:17.245933 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:17.245934    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k6hx\" (UniqueName: \"kubernetes.io/projected/7861e9eb-a526-442b-800f-6009fa55789a-kube-api-access-2k6hx\") pod \"payload-processing-e2e-worker-w2-f79c02-5c7b8dcbf7-qtwl6\" (UID: \"7861e9eb-a526-442b-800f-6009fa55789a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-f79c02-5c7b8dcbf7-qtwl6"
Nov 03 17:56:17.246714 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:17.246690    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f79c02-5477cd8fff-2h4nk"
Nov 03 17:56:17.347568 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:17.347532    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7861e9eb-a526-442b-800f-6009fa55789a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-f79c02-5c7b8dcbf7-qtwl6\" (UID: \"7861e9eb-a526-442b-800f-6009fa55789a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-f79c02-5c7b8dcbf7-qtwl6"
Nov 03 17:56:17.347568 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:17.347570    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k6hx\" (UniqueName: \"kubernetes.io/projected/7861e9eb-a526-442b-800f-6009fa55789a-kube-api-access-2k6hx\") pod \"payload-processing-e2e-worker-w2-f79c02-5c7b8dcbf7-qtwl6\" (UID: \"7861e9eb-a526-442b-800f-6009fa55789a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-f79c02-5c7b8dcbf7-qtwl6"
Nov 03 17:56:17.348362 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:17.348341    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7861e9eb-a526-442b-800f-6009fa55789a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-f79c02-5c7b8dcbf7-qtwl6\" (UID: \"7861e9eb-a526-442b-800f-6009fa55789a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-f79c02-5c7b8dcbf7-qtwl6"
Nov 03 17:56:17.407704 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:17.407671    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k6hx\" (UniqueName: \"kubernetes.io/projected/7861e9eb-a526-442b-800f-6009fa55789a-kube-api-access-2k6hx\") pod \"payload-processing-e2e-worker-w2-f79c02-5c7b8dcbf7-qtwl6\" (UID: \"7861e9eb-a526-442b-800f-6009fa55789a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-f79c02-5c7b8dcbf7-qtwl6"
Nov 03 17:56:17.446590 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:17.446561    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-f79c02-5c7b8dcbf7-qtwl6"
Nov 03 17:56:17.632937 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:17.632901    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-f79c02-5477cd8fff-2h4nk"]
Nov 03 17:56:17.639976 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:56:17.639948    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13fcfa33_03ce_4384_a431_edf17a1f2735.slice/crio-0979ea834b2da0709685b58a011305ba3a956c4f126c7cafdba7608f906064bb WatchSource:0}: Error finding container 0979ea834b2da0709685b58a011305ba3a956c4f126c7cafdba7608f906064bb: Status 404 returned error can't find the container with id 0979ea834b2da0709685b58a011305ba3a956c4f126c7cafdba7608f906064bb
Nov 03 17:56:17.818697 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:17.818661    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-f79c02-5c7b8dcbf7-qtwl6"]
Nov 03 17:56:17.826038 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:56:17.826004    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7861e9eb_a526_442b_800f_6009fa55789a.slice/crio-bbaf8e680ee250859dde02a01d2e13d78a08b2583f93b3c39e00a16935010d61 WatchSource:0}: Error finding container bbaf8e680ee250859dde02a01d2e13d78a08b2583f93b3c39e00a16935010d61: Status 404 returned error can't find the container with id bbaf8e680ee250859dde02a01d2e13d78a08b2583f93b3c39e00a16935010d61
Nov 03 17:56:18.411292 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:18.411255    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-f79c02-5c7b8dcbf7-qtwl6" event={"ID":"7861e9eb-a526-442b-800f-6009fa55789a","Type":"ContainerStarted","Data":"cb5b039bde5463d14872c292f2434ff1fc45bd3258d6b823d2919b57416e6042"}
Nov 03 17:56:18.411791 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:18.411313    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-f79c02-5c7b8dcbf7-qtwl6" event={"ID":"7861e9eb-a526-442b-800f-6009fa55789a","Type":"ContainerStarted","Data":"bbaf8e680ee250859dde02a01d2e13d78a08b2583f93b3c39e00a16935010d61"}
Nov 03 17:56:18.411791 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:18.411344    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-f79c02-5c7b8dcbf7-qtwl6"
Nov 03 17:56:18.412960 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:18.412932    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f79c02-5477cd8fff-2h4nk" event={"ID":"13fcfa33-03ce-4384-a431-edf17a1f2735","Type":"ContainerStarted","Data":"da8194e643108ce1d44254e047b5d88c793a05b242660947f1578a1fd2af28ba"}
Nov 03 17:56:18.413066 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:18.412985    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f79c02-5477cd8fff-2h4nk"
Nov 03 17:56:18.413066 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:18.413001    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f79c02-5477cd8fff-2h4nk" event={"ID":"13fcfa33-03ce-4384-a431-edf17a1f2735","Type":"ContainerStarted","Data":"0979ea834b2da0709685b58a011305ba3a956c4f126c7cafdba7608f906064bb"}
Nov 03 17:56:18.519811 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:18.519752    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-f79c02-5c7b8dcbf7-qtwl6" podStartSLOduration=1.5197377730000001 podStartE2EDuration="1.519737773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:18.519341469 +0000 UTC m=+617.535539538" watchObservedRunningTime="2026-11-03 17:56:18.519737773 +0000 UTC m=+617.535935842"
Nov 03 17:56:18.629171 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:18.629115    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f79c02-5477cd8fff-2h4nk" podStartSLOduration=2.6291005910000003 podStartE2EDuration="2.629100591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:18.627837793 +0000 UTC m=+617.644035861" watchObservedRunningTime="2026-11-03 17:56:18.629100591 +0000 UTC m=+617.645298707"
Nov 03 17:56:22.157318 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:22.157277    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-ef9633-769c9f5f8f-mj4br"]
Nov 03 17:56:22.160857 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:22.160833    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-ef9633-769c9f5f8f-mj4br"
Nov 03 17:56:22.197486 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:22.197450    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwh7k\" (UniqueName: \"kubernetes.io/projected/e8f65f51-9bb6-4242-a9d6-9a2934f06f84-kube-api-access-mwh7k\") pod \"payload-processing-e2e-worker-w1-ef9633-769c9f5f8f-mj4br\" (UID: \"e8f65f51-9bb6-4242-a9d6-9a2934f06f84\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-ef9633-769c9f5f8f-mj4br"
Nov 03 17:56:22.197655 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:22.197633    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8f65f51-9bb6-4242-a9d6-9a2934f06f84-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-ef9633-769c9f5f8f-mj4br\" (UID: \"e8f65f51-9bb6-4242-a9d6-9a2934f06f84\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-ef9633-769c9f5f8f-mj4br"
Nov 03 17:56:22.211963 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:22.211940    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-ef9633\""
Nov 03 17:56:22.257804 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:22.257774    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-ef9633-dockercfg-dcghn\""
Nov 03 17:56:22.259324 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:22.259299    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-ef9633-769c9f5f8f-mj4br"]
Nov 03 17:56:22.299184 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:22.299150    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8f65f51-9bb6-4242-a9d6-9a2934f06f84-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-ef9633-769c9f5f8f-mj4br\" (UID: \"e8f65f51-9bb6-4242-a9d6-9a2934f06f84\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-ef9633-769c9f5f8f-mj4br"
Nov 03 17:56:22.299372 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:22.299201    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwh7k\" (UniqueName: \"kubernetes.io/projected/e8f65f51-9bb6-4242-a9d6-9a2934f06f84-kube-api-access-mwh7k\") pod \"payload-processing-e2e-worker-w1-ef9633-769c9f5f8f-mj4br\" (UID: \"e8f65f51-9bb6-4242-a9d6-9a2934f06f84\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-ef9633-769c9f5f8f-mj4br"
Nov 03 17:56:22.299951 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:22.299924    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8f65f51-9bb6-4242-a9d6-9a2934f06f84-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-ef9633-769c9f5f8f-mj4br\" (UID: \"e8f65f51-9bb6-4242-a9d6-9a2934f06f84\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-ef9633-769c9f5f8f-mj4br"
Nov 03 17:56:22.356804 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:22.356770    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwh7k\" (UniqueName: \"kubernetes.io/projected/e8f65f51-9bb6-4242-a9d6-9a2934f06f84-kube-api-access-mwh7k\") pod \"payload-processing-e2e-worker-w1-ef9633-769c9f5f8f-mj4br\" (UID: \"e8f65f51-9bb6-4242-a9d6-9a2934f06f84\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-ef9633-769c9f5f8f-mj4br"
Nov 03 17:56:22.473093 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:22.473010    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-ef9633-769c9f5f8f-mj4br"
Nov 03 17:56:22.841309 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:22.841266    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-ef9633-769c9f5f8f-mj4br"]
Nov 03 17:56:22.849150 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:56:22.849122    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8f65f51_9bb6_4242_a9d6_9a2934f06f84.slice/crio-fe6c235ce2f3ffc9f03b6b506b5f8b7800edae5b4aafddb4f27350dc2932a1ef WatchSource:0}: Error finding container fe6c235ce2f3ffc9f03b6b506b5f8b7800edae5b4aafddb4f27350dc2932a1ef: Status 404 returned error can't find the container with id fe6c235ce2f3ffc9f03b6b506b5f8b7800edae5b4aafddb4f27350dc2932a1ef
Nov 03 17:56:23.436956 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:23.436916    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-ef9633-769c9f5f8f-mj4br" event={"ID":"e8f65f51-9bb6-4242-a9d6-9a2934f06f84","Type":"ContainerStarted","Data":"d03879416fa4f2f391ba7f5c5c3f6b85442a7cd46b8b6327ba8f76d6917947cb"}
Nov 03 17:56:23.437389 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:23.436969    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-ef9633-769c9f5f8f-mj4br" event={"ID":"e8f65f51-9bb6-4242-a9d6-9a2934f06f84","Type":"ContainerStarted","Data":"fe6c235ce2f3ffc9f03b6b506b5f8b7800edae5b4aafddb4f27350dc2932a1ef"}
Nov 03 17:56:23.437389 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:23.437047    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-ef9633-769c9f5f8f-mj4br"
Nov 03 17:56:23.546043 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:23.545990    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-ef9633-769c9f5f8f-mj4br" podStartSLOduration=1.5459773449999998 podStartE2EDuration="1.545977345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:23.54484662 +0000 UTC m=+622.561044689" watchObservedRunningTime="2026-11-03 17:56:23.545977345 +0000 UTC m=+622.562175414"
Nov 03 17:56:24.572515 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:24.572485    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-aab753-7bb466b675-bqngj"]
Nov 03 17:56:24.575092 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:24.575064    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aab753-7bb466b675-bqngj"
Nov 03 17:56:24.577388 ip-10-0-128-204 kubenswrapper[5315]: E1103 17:56:24.577363    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 17:56:24.618780 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:24.618744    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnwj5\" (UniqueName: \"kubernetes.io/projected/0f010c6f-b9a7-4367-874e-3412af41460a-kube-api-access-rnwj5\") pod \"payload-pre-processing-e2e-worker-w6-aab753-7bb466b675-bqngj\" (UID: \"0f010c6f-b9a7-4367-874e-3412af41460a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aab753-7bb466b675-bqngj"
Nov 03 17:56:24.618938 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:24.618816    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f010c6f-b9a7-4367-874e-3412af41460a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-aab753-7bb466b675-bqngj\" (UID: \"0f010c6f-b9a7-4367-874e-3412af41460a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aab753-7bb466b675-bqngj"
Nov 03 17:56:24.628977 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:24.624222    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-aab753\""
Nov 03 17:56:24.672285 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:24.672234    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-aab753-7bb466b675-bqngj"]
Nov 03 17:56:24.672472 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:24.672453    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-aab753-dockercfg-54xmm\""
Nov 03 17:56:24.720251 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:24.720214    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f010c6f-b9a7-4367-874e-3412af41460a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-aab753-7bb466b675-bqngj\" (UID: \"0f010c6f-b9a7-4367-874e-3412af41460a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aab753-7bb466b675-bqngj"
Nov 03 17:56:24.720408 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:24.720303    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnwj5\" (UniqueName: \"kubernetes.io/projected/0f010c6f-b9a7-4367-874e-3412af41460a-kube-api-access-rnwj5\") pod \"payload-pre-processing-e2e-worker-w6-aab753-7bb466b675-bqngj\" (UID: \"0f010c6f-b9a7-4367-874e-3412af41460a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aab753-7bb466b675-bqngj"
Nov 03 17:56:24.720954 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:24.720932    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f010c6f-b9a7-4367-874e-3412af41460a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-aab753-7bb466b675-bqngj\" (UID: \"0f010c6f-b9a7-4367-874e-3412af41460a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aab753-7bb466b675-bqngj"
Nov 03 17:56:24.780757 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:24.780720    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnwj5\" (UniqueName: \"kubernetes.io/projected/0f010c6f-b9a7-4367-874e-3412af41460a-kube-api-access-rnwj5\") pod \"payload-pre-processing-e2e-worker-w6-aab753-7bb466b675-bqngj\" (UID: \"0f010c6f-b9a7-4367-874e-3412af41460a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aab753-7bb466b675-bqngj"
Nov 03 17:56:24.894201 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:24.893668    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aab753-7bb466b675-bqngj"
Nov 03 17:56:25.264171 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:25.264090    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-aab753-7bb466b675-bqngj"]
Nov 03 17:56:25.270602 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:56:25.270572    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f010c6f_b9a7_4367_874e_3412af41460a.slice/crio-6038efba68d91282160c4d173c67ad2fd241313642234915b8dabf8f406f9876 WatchSource:0}: Error finding container 6038efba68d91282160c4d173c67ad2fd241313642234915b8dabf8f406f9876: Status 404 returned error can't find the container with id 6038efba68d91282160c4d173c67ad2fd241313642234915b8dabf8f406f9876
Nov 03 17:56:25.456605 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:25.456567    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aab753-7bb466b675-bqngj" event={"ID":"0f010c6f-b9a7-4367-874e-3412af41460a","Type":"ContainerStarted","Data":"1d8a1abb27f3003b5561b1517d2b000e5586fefe9fa1ab7b4b8ae640166de107"}
Nov 03 17:56:25.456779 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:25.456645    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aab753-7bb466b675-bqngj" event={"ID":"0f010c6f-b9a7-4367-874e-3412af41460a","Type":"ContainerStarted","Data":"6038efba68d91282160c4d173c67ad2fd241313642234915b8dabf8f406f9876"}
Nov 03 17:56:25.456779 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:25.456766    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aab753-7bb466b675-bqngj"
Nov 03 17:56:25.567142 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:25.567085    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aab753-7bb466b675-bqngj" podStartSLOduration=1.56707094 podStartE2EDuration="1.56707094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:25.566777041 +0000 UTC m=+624.582975110" watchObservedRunningTime="2026-11-03 17:56:25.56707094 +0000 UTC m=+624.583269001"
Nov 03 17:56:29.435635 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:29.430996    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f79c02-5477cd8fff-2h4nk"
Nov 03 17:56:29.436157 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:29.435768    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-f79c02-5c7b8dcbf7-qtwl6"
Nov 03 17:56:32.054141 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:32.054101    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-0ccbb9-6b4b8ccc7f-q47rc"]
Nov 03 17:56:32.056814 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:32.056783    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0ccbb9-6b4b8ccc7f-q47rc"
Nov 03 17:56:32.114050 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:32.114016    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7580e5e-cc8b-412e-99f0-3a71c762e618-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-0ccbb9-6b4b8ccc7f-q47rc\" (UID: \"e7580e5e-cc8b-412e-99f0-3a71c762e618\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0ccbb9-6b4b8ccc7f-q47rc"
Nov 03 17:56:32.114226 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:32.114072    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thwfg\" (UniqueName: \"kubernetes.io/projected/e7580e5e-cc8b-412e-99f0-3a71c762e618-kube-api-access-thwfg\") pod \"payload-pre-processing-e2e-worker-w0-0ccbb9-6b4b8ccc7f-q47rc\" (UID: \"e7580e5e-cc8b-412e-99f0-3a71c762e618\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0ccbb9-6b4b8ccc7f-q47rc"
Nov 03 17:56:32.151667 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:32.151632    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-0ccbb9\""
Nov 03 17:56:32.151667 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:32.151637    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-0ccbb9-dockercfg-xtmwv\""
Nov 03 17:56:32.153176 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:32.153155    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-0ccbb9-6b4b8ccc7f-q47rc"]
Nov 03 17:56:32.208139 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:32.208090    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-0ccbb9-57dcc5ff7-zcv75"]
Nov 03 17:56:32.211643 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:32.211602    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-0ccbb9-57dcc5ff7-zcv75"
Nov 03 17:56:32.214913 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:32.214877    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thwfg\" (UniqueName: \"kubernetes.io/projected/e7580e5e-cc8b-412e-99f0-3a71c762e618-kube-api-access-thwfg\") pod \"payload-pre-processing-e2e-worker-w0-0ccbb9-6b4b8ccc7f-q47rc\" (UID: \"e7580e5e-cc8b-412e-99f0-3a71c762e618\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0ccbb9-6b4b8ccc7f-q47rc"
Nov 03 17:56:32.215065 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:32.215029    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7580e5e-cc8b-412e-99f0-3a71c762e618-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-0ccbb9-6b4b8ccc7f-q47rc\" (UID: \"e7580e5e-cc8b-412e-99f0-3a71c762e618\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0ccbb9-6b4b8ccc7f-q47rc"
Nov 03 17:56:32.224452 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:32.224422    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7580e5e-cc8b-412e-99f0-3a71c762e618-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-0ccbb9-6b4b8ccc7f-q47rc\" (UID: \"e7580e5e-cc8b-412e-99f0-3a71c762e618\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0ccbb9-6b4b8ccc7f-q47rc"
Nov 03 17:56:32.274028 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:32.274001    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-0ccbb9-57dcc5ff7-zcv75"]
Nov 03 17:56:32.283036 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:32.283007    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thwfg\" (UniqueName: \"kubernetes.io/projected/e7580e5e-cc8b-412e-99f0-3a71c762e618-kube-api-access-thwfg\") pod \"payload-pre-processing-e2e-worker-w0-0ccbb9-6b4b8ccc7f-q47rc\" (UID: \"e7580e5e-cc8b-412e-99f0-3a71c762e618\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0ccbb9-6b4b8ccc7f-q47rc"
Nov 03 17:56:32.316566 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:32.316486    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b86eb9eb-969a-4ac5-a947-0f1d32ca5b72-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-0ccbb9-57dcc5ff7-zcv75\" (UID: \"b86eb9eb-969a-4ac5-a947-0f1d32ca5b72\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-0ccbb9-57dcc5ff7-zcv75"
Nov 03 17:56:32.316699 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:32.316651    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dln4\" (UniqueName: \"kubernetes.io/projected/b86eb9eb-969a-4ac5-a947-0f1d32ca5b72-kube-api-access-4dln4\") pod \"payload-processing-e2e-worker-w0-0ccbb9-57dcc5ff7-zcv75\" (UID: \"b86eb9eb-969a-4ac5-a947-0f1d32ca5b72\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-0ccbb9-57dcc5ff7-zcv75"
Nov 03 17:56:32.369138 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:32.369101    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0ccbb9-6b4b8ccc7f-q47rc"
Nov 03 17:56:32.417906 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:32.417720    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b86eb9eb-969a-4ac5-a947-0f1d32ca5b72-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-0ccbb9-57dcc5ff7-zcv75\" (UID: \"b86eb9eb-969a-4ac5-a947-0f1d32ca5b72\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-0ccbb9-57dcc5ff7-zcv75"
Nov 03 17:56:32.417998 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:32.417918    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dln4\" (UniqueName: \"kubernetes.io/projected/b86eb9eb-969a-4ac5-a947-0f1d32ca5b72-kube-api-access-4dln4\") pod \"payload-processing-e2e-worker-w0-0ccbb9-57dcc5ff7-zcv75\" (UID: \"b86eb9eb-969a-4ac5-a947-0f1d32ca5b72\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-0ccbb9-57dcc5ff7-zcv75"
Nov 03 17:56:32.418870 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:32.418843    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b86eb9eb-969a-4ac5-a947-0f1d32ca5b72-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-0ccbb9-57dcc5ff7-zcv75\" (UID: \"b86eb9eb-969a-4ac5-a947-0f1d32ca5b72\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-0ccbb9-57dcc5ff7-zcv75"
Nov 03 17:56:32.475471 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:32.475440    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dln4\" (UniqueName: \"kubernetes.io/projected/b86eb9eb-969a-4ac5-a947-0f1d32ca5b72-kube-api-access-4dln4\") pod \"payload-processing-e2e-worker-w0-0ccbb9-57dcc5ff7-zcv75\" (UID: \"b86eb9eb-969a-4ac5-a947-0f1d32ca5b72\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-0ccbb9-57dcc5ff7-zcv75"
Nov 03 17:56:32.527955 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:32.527922    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-0ccbb9-57dcc5ff7-zcv75"
Nov 03 17:56:32.740495 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:32.740453    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-0ccbb9-6b4b8ccc7f-q47rc"]
Nov 03 17:56:32.747474 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:56:32.747448    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7580e5e_cc8b_412e_99f0_3a71c762e618.slice/crio-15e6d9e21da3a3bbb398139f46f0d8827b4a38ed4703ed0ef5df8c1e347cf7c7 WatchSource:0}: Error finding container 15e6d9e21da3a3bbb398139f46f0d8827b4a38ed4703ed0ef5df8c1e347cf7c7: Status 404 returned error can't find the container with id 15e6d9e21da3a3bbb398139f46f0d8827b4a38ed4703ed0ef5df8c1e347cf7c7
Nov 03 17:56:32.901728 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:32.901641    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-0ccbb9-57dcc5ff7-zcv75"]
Nov 03 17:56:32.910274 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:56:32.910236    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb86eb9eb_969a_4ac5_a947_0f1d32ca5b72.slice/crio-16bbff5255624217845bcc0284d39dae1f8c28554a996962d852a1a1608b93da WatchSource:0}: Error finding container 16bbff5255624217845bcc0284d39dae1f8c28554a996962d852a1a1608b93da: Status 404 returned error can't find the container with id 16bbff5255624217845bcc0284d39dae1f8c28554a996962d852a1a1608b93da
Nov 03 17:56:33.493830 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:33.493768    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-0ccbb9-57dcc5ff7-zcv75" event={"ID":"b86eb9eb-969a-4ac5-a947-0f1d32ca5b72","Type":"ContainerStarted","Data":"739bb1e5c2fba8967292d6a0db43d1eda8dfcd7af96cf7a2719ff6ca72202205"}
Nov 03 17:56:33.494316 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:33.493885    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-0ccbb9-57dcc5ff7-zcv75" event={"ID":"b86eb9eb-969a-4ac5-a947-0f1d32ca5b72","Type":"ContainerStarted","Data":"16bbff5255624217845bcc0284d39dae1f8c28554a996962d852a1a1608b93da"}
Nov 03 17:56:33.494316 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:33.493951    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-0ccbb9-57dcc5ff7-zcv75"
Nov 03 17:56:33.502409 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:33.502374    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0ccbb9-6b4b8ccc7f-q47rc" event={"ID":"e7580e5e-cc8b-412e-99f0-3a71c762e618","Type":"ContainerStarted","Data":"4f586d9a86d82eca3d2d65bae3fae862f698adcdf5d24d6628fd2d81bbc45a69"}
Nov 03 17:56:33.502596 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:33.502427    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0ccbb9-6b4b8ccc7f-q47rc" event={"ID":"e7580e5e-cc8b-412e-99f0-3a71c762e618","Type":"ContainerStarted","Data":"15e6d9e21da3a3bbb398139f46f0d8827b4a38ed4703ed0ef5df8c1e347cf7c7"}
Nov 03 17:56:33.502596 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:33.502473    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0ccbb9-6b4b8ccc7f-q47rc"
Nov 03 17:56:33.604167 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:33.604105    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-0ccbb9-57dcc5ff7-zcv75" podStartSLOduration=1.604091692 podStartE2EDuration="1.604091692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:33.602434304 +0000 UTC m=+632.618632375" watchObservedRunningTime="2026-11-03 17:56:33.604091692 +0000 UTC m=+632.620289760"
Nov 03 17:56:33.711364 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:33.711304    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0ccbb9-6b4b8ccc7f-q47rc" podStartSLOduration=1.711289917 podStartE2EDuration="1.711289917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:33.710784522 +0000 UTC m=+632.726982591" watchObservedRunningTime="2026-11-03 17:56:33.711289917 +0000 UTC m=+632.727487985"
Nov 03 17:56:34.454223 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:34.454189    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-ef9633-769c9f5f8f-mj4br"
Nov 03 17:56:36.463283 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:36.463248    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aab753-7bb466b675-bqngj"
Nov 03 17:56:36.517222 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:36.517192    5315 generic.go:356] "Generic (PLEG): container finished" podID="dd5c9fcf-e132-42e2-b031-c608cfd0b392" containerID="f2a6ea1f91953839e03ec6653e0e9e5ef8b0e68ec3835cf1c567f56ff182207a" exitCode=0
Nov 03 17:56:36.517382 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:36.517242    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855" event={"ID":"dd5c9fcf-e132-42e2-b031-c608cfd0b392","Type":"ContainerDied","Data":"f2a6ea1f91953839e03ec6653e0e9e5ef8b0e68ec3835cf1c567f56ff182207a"}
Nov 03 17:56:36.517601 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:36.517587    5315 scope.go:122] "RemoveContainer" containerID="f2a6ea1f91953839e03ec6653e0e9e5ef8b0e68ec3835cf1c567f56ff182207a"
Nov 03 17:56:36.518914 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:36.518883    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 17:56:36.519034 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:36.518964    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 17:56:36.519034 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:36.519015    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 17:56:37.106650 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:37.105890    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-0adcbc-b4854959-pwnvs"]
Nov 03 17:56:37.111509 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:37.109855    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0adcbc-b4854959-pwnvs"
Nov 03 17:56:37.161635 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:37.159647    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-0adcbc\""
Nov 03 17:56:37.161635 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:37.159774    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-0adcbc-dockercfg-sqblj\""
Nov 03 17:56:37.169636 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:37.166732    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-0adcbc-b4854959-pwnvs"]
Nov 03 17:56:37.181647 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:37.177133    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssvsr\" (UniqueName: \"kubernetes.io/projected/1120dc4d-3916-41fc-99b4-3c91ff69f28e-kube-api-access-ssvsr\") pod \"payload-pre-processing-e2e-worker-w3-0adcbc-b4854959-pwnvs\" (UID: \"1120dc4d-3916-41fc-99b4-3c91ff69f28e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0adcbc-b4854959-pwnvs"
Nov 03 17:56:37.181647 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:37.177224    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1120dc4d-3916-41fc-99b4-3c91ff69f28e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-0adcbc-b4854959-pwnvs\" (UID: \"1120dc4d-3916-41fc-99b4-3c91ff69f28e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0adcbc-b4854959-pwnvs"
Nov 03 17:56:37.243633 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:37.242258    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-0adcbc-695b8c899-b9rjk"]
Nov 03 17:56:37.262632 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:37.260977    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-0adcbc-695b8c899-b9rjk"
Nov 03 17:56:37.281843 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:37.280512    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssvsr\" (UniqueName: \"kubernetes.io/projected/1120dc4d-3916-41fc-99b4-3c91ff69f28e-kube-api-access-ssvsr\") pod \"payload-pre-processing-e2e-worker-w3-0adcbc-b4854959-pwnvs\" (UID: \"1120dc4d-3916-41fc-99b4-3c91ff69f28e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0adcbc-b4854959-pwnvs"
Nov 03 17:56:37.286654 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:37.286398    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1120dc4d-3916-41fc-99b4-3c91ff69f28e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-0adcbc-b4854959-pwnvs\" (UID: \"1120dc4d-3916-41fc-99b4-3c91ff69f28e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0adcbc-b4854959-pwnvs"
Nov 03 17:56:37.286826 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:37.280597    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1120dc4d-3916-41fc-99b4-3c91ff69f28e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-0adcbc-b4854959-pwnvs\" (UID: \"1120dc4d-3916-41fc-99b4-3c91ff69f28e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0adcbc-b4854959-pwnvs"
Nov 03 17:56:37.296009 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:37.295986    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-0adcbc-695b8c899-b9rjk"]
Nov 03 17:56:37.382361 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:37.382244    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssvsr\" (UniqueName: \"kubernetes.io/projected/1120dc4d-3916-41fc-99b4-3c91ff69f28e-kube-api-access-ssvsr\") pod \"payload-pre-processing-e2e-worker-w3-0adcbc-b4854959-pwnvs\" (UID: \"1120dc4d-3916-41fc-99b4-3c91ff69f28e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0adcbc-b4854959-pwnvs"
Nov 03 17:56:37.387750 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:37.387724    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fa9bc33-97f0-40eb-aee3-f2a6923736a1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-0adcbc-695b8c899-b9rjk\" (UID: \"0fa9bc33-97f0-40eb-aee3-f2a6923736a1\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-0adcbc-695b8c899-b9rjk"
Nov 03 17:56:37.387953 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:37.387937    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snq2h\" (UniqueName: \"kubernetes.io/projected/0fa9bc33-97f0-40eb-aee3-f2a6923736a1-kube-api-access-snq2h\") pod \"payload-processing-e2e-worker-w3-0adcbc-695b8c899-b9rjk\" (UID: \"0fa9bc33-97f0-40eb-aee3-f2a6923736a1\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-0adcbc-695b8c899-b9rjk"
Nov 03 17:56:37.445633 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:37.444791    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0adcbc-b4854959-pwnvs"
Nov 03 17:56:37.495814 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:37.489600    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fa9bc33-97f0-40eb-aee3-f2a6923736a1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-0adcbc-695b8c899-b9rjk\" (UID: \"0fa9bc33-97f0-40eb-aee3-f2a6923736a1\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-0adcbc-695b8c899-b9rjk"
Nov 03 17:56:37.495814 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:37.489742    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snq2h\" (UniqueName: \"kubernetes.io/projected/0fa9bc33-97f0-40eb-aee3-f2a6923736a1-kube-api-access-snq2h\") pod \"payload-processing-e2e-worker-w3-0adcbc-695b8c899-b9rjk\" (UID: \"0fa9bc33-97f0-40eb-aee3-f2a6923736a1\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-0adcbc-695b8c899-b9rjk"
Nov 03 17:56:37.495814 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:37.491036    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fa9bc33-97f0-40eb-aee3-f2a6923736a1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-0adcbc-695b8c899-b9rjk\" (UID: \"0fa9bc33-97f0-40eb-aee3-f2a6923736a1\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-0adcbc-695b8c899-b9rjk"
Nov 03 17:56:37.535645 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:37.534260    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855" event={"ID":"dd5c9fcf-e132-42e2-b031-c608cfd0b392","Type":"ContainerStarted","Data":"3808d426e0bdb3213501e57a43960bc2d0e77c720136cd88efdf64fcd9872873"}
Nov 03 17:56:37.556334 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:37.556236    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855"
Nov 03 17:56:37.572891 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:37.570754    5315 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Nov 03 17:56:37.572891 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:37.570826    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855" podUID="dd5c9fcf-e132-42e2-b031-c608cfd0b392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Nov 03 17:56:37.572891 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:37.571784    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snq2h\" (UniqueName: \"kubernetes.io/projected/0fa9bc33-97f0-40eb-aee3-f2a6923736a1-kube-api-access-snq2h\") pod \"payload-processing-e2e-worker-w3-0adcbc-695b8c899-b9rjk\" (UID: \"0fa9bc33-97f0-40eb-aee3-f2a6923736a1\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-0adcbc-695b8c899-b9rjk"
Nov 03 17:56:37.588127 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:37.588101    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-0adcbc-695b8c899-b9rjk"
Nov 03 17:56:37.906824 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:37.906791    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-0adcbc-b4854959-pwnvs"]
Nov 03 17:56:37.914964 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:56:37.914939    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1120dc4d_3916_41fc_99b4_3c91ff69f28e.slice/crio-4ca0ddfd5f3539c067683518fcfe1eff5bb7c255d6bb5941be8e51bea5af2736 WatchSource:0}: Error finding container 4ca0ddfd5f3539c067683518fcfe1eff5bb7c255d6bb5941be8e51bea5af2736: Status 404 returned error can't find the container with id 4ca0ddfd5f3539c067683518fcfe1eff5bb7c255d6bb5941be8e51bea5af2736
Nov 03 17:56:38.013600 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:38.013574    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-0adcbc-695b8c899-b9rjk"]
Nov 03 17:56:38.021887 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:56:38.021853    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fa9bc33_97f0_40eb_aee3_f2a6923736a1.slice/crio-f2498e8410ca3f7eceafb27c904828afd6a30d115d5c4acc9f9646a2bbd213b5 WatchSource:0}: Error finding container f2498e8410ca3f7eceafb27c904828afd6a30d115d5c4acc9f9646a2bbd213b5: Status 404 returned error can't find the container with id f2498e8410ca3f7eceafb27c904828afd6a30d115d5c4acc9f9646a2bbd213b5
Nov 03 17:56:38.570642 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:38.555824    5315 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Nov 03 17:56:38.572700 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:38.572657    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855" podUID="dd5c9fcf-e132-42e2-b031-c608cfd0b392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Nov 03 17:56:38.575233 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:38.575018    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-0adcbc-695b8c899-b9rjk" event={"ID":"0fa9bc33-97f0-40eb-aee3-f2a6923736a1","Type":"ContainerStarted","Data":"fc5e8915c7cb2bf3e174c352cbe61e150bfd3ed046ea85bf4309f4fba8df2739"}
Nov 03 17:56:38.575233 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:38.575225    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-0adcbc-695b8c899-b9rjk"
Nov 03 17:56:38.575465 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:38.575262    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-0adcbc-695b8c899-b9rjk" event={"ID":"0fa9bc33-97f0-40eb-aee3-f2a6923736a1","Type":"ContainerStarted","Data":"f2498e8410ca3f7eceafb27c904828afd6a30d115d5c4acc9f9646a2bbd213b5"}
Nov 03 17:56:38.577790 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:38.577764    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0adcbc-b4854959-pwnvs" event={"ID":"1120dc4d-3916-41fc-99b4-3c91ff69f28e","Type":"ContainerStarted","Data":"5c2775dd53a7b83d1599b3c4d2042d589d9d72bb66ef5047e99a05378e6638b1"}
Nov 03 17:56:38.577914 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:38.577812    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0adcbc-b4854959-pwnvs" event={"ID":"1120dc4d-3916-41fc-99b4-3c91ff69f28e","Type":"ContainerStarted","Data":"4ca0ddfd5f3539c067683518fcfe1eff5bb7c255d6bb5941be8e51bea5af2736"}
Nov 03 17:56:38.577914 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:38.577839    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0adcbc-b4854959-pwnvs"
Nov 03 17:56:38.706184 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:38.706136    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-0adcbc-695b8c899-b9rjk" podStartSLOduration=1.706120984 podStartE2EDuration="1.706120984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:38.70456342 +0000 UTC m=+637.720761489" watchObservedRunningTime="2026-11-03 17:56:38.706120984 +0000 UTC m=+637.722319053"
Nov 03 17:56:38.834686 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:38.831221    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0adcbc-b4854959-pwnvs" podStartSLOduration=1.831201161 podStartE2EDuration="1.831201161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:38.830077728 +0000 UTC m=+637.846275799" watchObservedRunningTime="2026-11-03 17:56:38.831201161 +0000 UTC m=+637.847399230"
Nov 03 17:56:39.551635 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:39.550969    5315 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Nov 03 17:56:39.551635 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:39.551033    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855" podUID="dd5c9fcf-e132-42e2-b031-c608cfd0b392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Nov 03 17:56:40.377841 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:40.377794    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855"
Nov 03 17:56:40.566635 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:40.565782    5315 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Nov 03 17:56:40.566635 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:40.565896    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855" podUID="dd5c9fcf-e132-42e2-b031-c608cfd0b392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Nov 03 17:56:41.550156 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:41.550115    5315 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Nov 03 17:56:41.550540 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:41.550179    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855" podUID="dd5c9fcf-e132-42e2-b031-c608cfd0b392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Nov 03 17:56:42.346783 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:42.346680    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-a86228-f656db785-jcvfr"]
Nov 03 17:56:42.379656 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:42.363761    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-a86228-f656db785-jcvfr"
Nov 03 17:56:42.403176 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:42.403141    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-a86228-f656db785-jcvfr"]
Nov 03 17:56:42.450011 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:42.449945    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-a86228\""
Nov 03 17:56:42.458906 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:42.458833    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-a86228-dockercfg-54jbg\""
Nov 03 17:56:42.484469 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:42.484438    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cf6q\" (UniqueName: \"kubernetes.io/projected/6434783b-1272-4ccd-b5c4-5a7aeeab58c7-kube-api-access-2cf6q\") pod \"payload-processing-e2e-worker-w4-a86228-f656db785-jcvfr\" (UID: \"6434783b-1272-4ccd-b5c4-5a7aeeab58c7\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-a86228-f656db785-jcvfr"
Nov 03 17:56:42.484631 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:42.484489    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6434783b-1272-4ccd-b5c4-5a7aeeab58c7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-a86228-f656db785-jcvfr\" (UID: \"6434783b-1272-4ccd-b5c4-5a7aeeab58c7\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-a86228-f656db785-jcvfr"
Nov 03 17:56:42.554834 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:42.554757    5315 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Nov 03 17:56:42.554834 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:42.554833    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855" podUID="dd5c9fcf-e132-42e2-b031-c608cfd0b392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Nov 03 17:56:42.585447 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:42.585405    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cf6q\" (UniqueName: \"kubernetes.io/projected/6434783b-1272-4ccd-b5c4-5a7aeeab58c7-kube-api-access-2cf6q\") pod \"payload-processing-e2e-worker-w4-a86228-f656db785-jcvfr\" (UID: \"6434783b-1272-4ccd-b5c4-5a7aeeab58c7\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-a86228-f656db785-jcvfr"
Nov 03 17:56:42.585654 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:42.585477    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6434783b-1272-4ccd-b5c4-5a7aeeab58c7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-a86228-f656db785-jcvfr\" (UID: \"6434783b-1272-4ccd-b5c4-5a7aeeab58c7\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-a86228-f656db785-jcvfr"
Nov 03 17:56:42.586219 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:42.586198    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6434783b-1272-4ccd-b5c4-5a7aeeab58c7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-a86228-f656db785-jcvfr\" (UID: \"6434783b-1272-4ccd-b5c4-5a7aeeab58c7\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-a86228-f656db785-jcvfr"
Nov 03 17:56:42.643932 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:42.643860    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cf6q\" (UniqueName: \"kubernetes.io/projected/6434783b-1272-4ccd-b5c4-5a7aeeab58c7-kube-api-access-2cf6q\") pod \"payload-processing-e2e-worker-w4-a86228-f656db785-jcvfr\" (UID: \"6434783b-1272-4ccd-b5c4-5a7aeeab58c7\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-a86228-f656db785-jcvfr"
Nov 03 17:56:42.694797 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:42.694755    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-a86228-f656db785-jcvfr"
Nov 03 17:56:43.115152 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:43.115113    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-a86228-f656db785-jcvfr"]
Nov 03 17:56:43.122989 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:56:43.122960    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6434783b_1272_4ccd_b5c4_5a7aeeab58c7.slice/crio-ed6844c0ddc724170ee8a8bfff02c1402ed187e9e8752f51be95b0b062dc08f2 WatchSource:0}: Error finding container ed6844c0ddc724170ee8a8bfff02c1402ed187e9e8752f51be95b0b062dc08f2: Status 404 returned error can't find the container with id ed6844c0ddc724170ee8a8bfff02c1402ed187e9e8752f51be95b0b062dc08f2
Nov 03 17:56:43.554950 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:43.554906    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855"
Nov 03 17:56:43.556251 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:43.556226    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855"
Nov 03 17:56:43.609020 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:43.608974    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-a86228-f656db785-jcvfr" event={"ID":"6434783b-1272-4ccd-b5c4-5a7aeeab58c7","Type":"ContainerStarted","Data":"4f62f10b9e6389ae06381c234bdf4fa35ab045d5a47f6f0aef6e0e65fce8599e"}
Nov 03 17:56:43.609206 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:43.609037    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-a86228-f656db785-jcvfr" event={"ID":"6434783b-1272-4ccd-b5c4-5a7aeeab58c7","Type":"ContainerStarted","Data":"ed6844c0ddc724170ee8a8bfff02c1402ed187e9e8752f51be95b0b062dc08f2"}
Nov 03 17:56:43.609206 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:43.609069    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-a86228-f656db785-jcvfr"
Nov 03 17:56:43.898266 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:43.898159    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-a86228-f656db785-jcvfr" podStartSLOduration=1.89814024 podStartE2EDuration="1.89814024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:43.897345857 +0000 UTC m=+642.913543928" watchObservedRunningTime="2026-11-03 17:56:43.89814024 +0000 UTC m=+642.914338311"
Nov 03 17:56:44.509521 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:44.509491    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0ccbb9-6b4b8ccc7f-q47rc"
Nov 03 17:56:44.509744 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:44.509573    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-0ccbb9-57dcc5ff7-zcv75"
Nov 03 17:56:45.619945 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:45.619912    5315 generic.go:356] "Generic (PLEG): container finished" podID="4e8997a5-bfb9-4517-bb5e-632f8639e44d" containerID="6afcb66d3e49bf9e99b751a2306aa56d55b92c13a671ac2d80588a967d564ce2" exitCode=0
Nov 03 17:56:45.620442 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:45.619982    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45" event={"ID":"4e8997a5-bfb9-4517-bb5e-632f8639e44d","Type":"ContainerDied","Data":"6afcb66d3e49bf9e99b751a2306aa56d55b92c13a671ac2d80588a967d564ce2"}
Nov 03 17:56:45.620442 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:45.620362    5315 scope.go:122] "RemoveContainer" containerID="6afcb66d3e49bf9e99b751a2306aa56d55b92c13a671ac2d80588a967d564ce2"
Nov 03 17:56:45.622666 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:45.622635    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 17:56:45.622959 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:45.622932    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 17:56:45.623120 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:45.623102    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 17:56:46.650604 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:46.650566    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45" event={"ID":"4e8997a5-bfb9-4517-bb5e-632f8639e44d","Type":"ContainerStarted","Data":"2975927156f1c80407a69ee06540379ec4bdcc7d0e458453e8d773bdaaa70515"}
Nov 03 17:56:47.268082 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:47.268041    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45"
Nov 03 17:56:47.268258 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:47.268234    5315 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 03 17:56:47.268304 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:47.268291    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45" podUID="4e8997a5-bfb9-4517-bb5e-632f8639e44d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 03 17:56:48.267671 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:48.267631    5315 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 03 17:56:48.268156 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:48.267691    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45" podUID="4e8997a5-bfb9-4517-bb5e-632f8639e44d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 03 17:56:49.267999 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:49.267962    5315 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 03 17:56:49.268379 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:49.268029    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45" podUID="4e8997a5-bfb9-4517-bb5e-632f8639e44d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 03 17:56:49.586787 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:49.586756    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0adcbc-b4854959-pwnvs"
Nov 03 17:56:49.586981 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:49.586836    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-0adcbc-695b8c899-b9rjk"
Nov 03 17:56:50.268230 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:50.268183    5315 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 03 17:56:50.268772 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:50.268238    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45" podUID="4e8997a5-bfb9-4517-bb5e-632f8639e44d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 03 17:56:51.267790 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:51.267759    5315 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 03 17:56:51.267962 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:51.267811    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45" podUID="4e8997a5-bfb9-4517-bb5e-632f8639e44d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 03 17:56:52.272574 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:52.272540    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45"
Nov 03 17:56:52.273062 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:52.272839    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45"
Nov 03 17:56:52.273573 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:52.273554    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45"
Nov 03 17:56:54.578662 ip-10-0-128-204 kubenswrapper[5315]: E1103 17:56:54.578602    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 17:56:54.616650 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:56:54.616602    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-a86228-f656db785-jcvfr"
Nov 03 17:57:09.366395 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:09.366318    5315 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 03 17:57:09.366395 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:09.366374    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn" podUID="cb78e2af-5c83-4d6d-a562-274b12209c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 03 17:57:09.810433 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:09.810395    5315 generic.go:356] "Generic (PLEG): container finished" podID="cb78e2af-5c83-4d6d-a562-274b12209c20" containerID="ec2b17e223560519400e4f7691fa64d88d00bbd8d54624bfb48290322d244893" exitCode=0
Nov 03 17:57:09.810652 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:09.810463    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn" event={"ID":"cb78e2af-5c83-4d6d-a562-274b12209c20","Type":"ContainerDied","Data":"ec2b17e223560519400e4f7691fa64d88d00bbd8d54624bfb48290322d244893"}
Nov 03 17:57:09.810894 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:09.810877    5315 scope.go:122] "RemoveContainer" containerID="ec2b17e223560519400e4f7691fa64d88d00bbd8d54624bfb48290322d244893"
Nov 03 17:57:09.812156 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:09.812126    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 17:57:09.812241 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:09.812190    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 17:57:09.812241 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:09.812219    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 17:57:10.817626 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:10.817576    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn" event={"ID":"cb78e2af-5c83-4d6d-a562-274b12209c20","Type":"ContainerStarted","Data":"6da1959a991a4e6d878dc7df2369f8a6a53f12464d9442d98ed7737ca80b9eef"}
Nov 03 17:57:11.249147 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:11.249047    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn"
Nov 03 17:57:11.249523 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:11.249427    5315 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 03 17:57:11.249743 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:11.249718    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn" podUID="cb78e2af-5c83-4d6d-a562-274b12209c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 03 17:57:11.921441 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:11.921411    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7"]
Nov 03 17:57:11.956992 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:11.956962    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7"
Nov 03 17:57:11.993674 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:11.993561    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7"]
Nov 03 17:57:12.064099 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:12.064061    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-aab753\"/\"default-dockercfg-b6977\""
Nov 03 17:57:12.064670 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:12.064647    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-aab753\"/\"simulator-w6-aab753-kserve-self-signed-certs\""
Nov 03 17:57:12.065113 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:12.065082    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-aab753\"/\"kube-root-ca.crt\""
Nov 03 17:57:12.065418 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:12.065402    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-aab753\"/\"openshift-service-ca.crt\""
Nov 03 17:57:12.101099 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:12.101063    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33725a51-6459-4f15-a866-e9dee41a9732-tls-certs\") pod \"simulator-w6-aab753-kserve-79c4b67bbb-h6rx7\" (UID: \"33725a51-6459-4f15-a866-e9dee41a9732\") " pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7"
Nov 03 17:57:12.101272 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:12.101136    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33725a51-6459-4f15-a866-e9dee41a9732-home\") pod \"simulator-w6-aab753-kserve-79c4b67bbb-h6rx7\" (UID: \"33725a51-6459-4f15-a866-e9dee41a9732\") " pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7"
Nov 03 17:57:12.101272 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:12.101171    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33725a51-6459-4f15-a866-e9dee41a9732-model-cache\") pod \"simulator-w6-aab753-kserve-79c4b67bbb-h6rx7\" (UID: \"33725a51-6459-4f15-a866-e9dee41a9732\") " pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7"
Nov 03 17:57:12.101272 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:12.101201    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33725a51-6459-4f15-a866-e9dee41a9732-tmp-dir\") pod \"simulator-w6-aab753-kserve-79c4b67bbb-h6rx7\" (UID: \"33725a51-6459-4f15-a866-e9dee41a9732\") " pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7"
Nov 03 17:57:12.101377 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:12.101288    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blvks\" (UniqueName: \"kubernetes.io/projected/33725a51-6459-4f15-a866-e9dee41a9732-kube-api-access-blvks\") pod \"simulator-w6-aab753-kserve-79c4b67bbb-h6rx7\" (UID: \"33725a51-6459-4f15-a866-e9dee41a9732\") " pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7"
Nov 03 17:57:12.101377 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:12.101342    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33725a51-6459-4f15-a866-e9dee41a9732-dshm\") pod \"simulator-w6-aab753-kserve-79c4b67bbb-h6rx7\" (UID: \"33725a51-6459-4f15-a866-e9dee41a9732\") " pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7"
Nov 03 17:57:12.202944 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:12.202861    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blvks\" (UniqueName: \"kubernetes.io/projected/33725a51-6459-4f15-a866-e9dee41a9732-kube-api-access-blvks\") pod \"simulator-w6-aab753-kserve-79c4b67bbb-h6rx7\" (UID: \"33725a51-6459-4f15-a866-e9dee41a9732\") " pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7"
Nov 03 17:57:12.202944 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:12.202909    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33725a51-6459-4f15-a866-e9dee41a9732-dshm\") pod \"simulator-w6-aab753-kserve-79c4b67bbb-h6rx7\" (UID: \"33725a51-6459-4f15-a866-e9dee41a9732\") " pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7"
Nov 03 17:57:12.203128 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:12.202958    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33725a51-6459-4f15-a866-e9dee41a9732-tls-certs\") pod \"simulator-w6-aab753-kserve-79c4b67bbb-h6rx7\" (UID: \"33725a51-6459-4f15-a866-e9dee41a9732\") " pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7"
Nov 03 17:57:12.203128 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:12.202989    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33725a51-6459-4f15-a866-e9dee41a9732-home\") pod \"simulator-w6-aab753-kserve-79c4b67bbb-h6rx7\" (UID: \"33725a51-6459-4f15-a866-e9dee41a9732\") " pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7"
Nov 03 17:57:12.203128 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:12.203008    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33725a51-6459-4f15-a866-e9dee41a9732-model-cache\") pod \"simulator-w6-aab753-kserve-79c4b67bbb-h6rx7\" (UID: \"33725a51-6459-4f15-a866-e9dee41a9732\") " pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7"
Nov 03 17:57:12.203128 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:12.203042    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33725a51-6459-4f15-a866-e9dee41a9732-tmp-dir\") pod \"simulator-w6-aab753-kserve-79c4b67bbb-h6rx7\" (UID: \"33725a51-6459-4f15-a866-e9dee41a9732\") " pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7"
Nov 03 17:57:12.203409 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:12.203393    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33725a51-6459-4f15-a866-e9dee41a9732-tmp-dir\") pod \"simulator-w6-aab753-kserve-79c4b67bbb-h6rx7\" (UID: \"33725a51-6459-4f15-a866-e9dee41a9732\") " pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7"
Nov 03 17:57:12.203578 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:12.203565    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33725a51-6459-4f15-a866-e9dee41a9732-home\") pod \"simulator-w6-aab753-kserve-79c4b67bbb-h6rx7\" (UID: \"33725a51-6459-4f15-a866-e9dee41a9732\") " pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7"
Nov 03 17:57:12.203866 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:12.203849    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33725a51-6459-4f15-a866-e9dee41a9732-model-cache\") pod \"simulator-w6-aab753-kserve-79c4b67bbb-h6rx7\" (UID: \"33725a51-6459-4f15-a866-e9dee41a9732\") " pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7"
Nov 03 17:57:12.205436 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:12.205418    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33725a51-6459-4f15-a866-e9dee41a9732-tls-certs\") pod \"simulator-w6-aab753-kserve-79c4b67bbb-h6rx7\" (UID: \"33725a51-6459-4f15-a866-e9dee41a9732\") " pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7"
Nov 03 17:57:12.206217 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:12.206199    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33725a51-6459-4f15-a866-e9dee41a9732-dshm\") pod \"simulator-w6-aab753-kserve-79c4b67bbb-h6rx7\" (UID: \"33725a51-6459-4f15-a866-e9dee41a9732\") " pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7"
Nov 03 17:57:12.248078 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:12.248040    5315 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 03 17:57:12.248396 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:12.248361    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn" podUID="cb78e2af-5c83-4d6d-a562-274b12209c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 03 17:57:12.265640 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:12.264323    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blvks\" (UniqueName: \"kubernetes.io/projected/33725a51-6459-4f15-a866-e9dee41a9732-kube-api-access-blvks\") pod \"simulator-w6-aab753-kserve-79c4b67bbb-h6rx7\" (UID: \"33725a51-6459-4f15-a866-e9dee41a9732\") " pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7"
Nov 03 17:57:12.295996 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:12.295957    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7"
Nov 03 17:57:12.746668 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:12.746589    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7"]
Nov 03 17:57:12.881647 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:12.879730    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7" event={"ID":"33725a51-6459-4f15-a866-e9dee41a9732","Type":"ContainerStarted","Data":"f615d5f66f36b3600db5973901433e72ea6cf322a41c20ee33c0772f7662e910"}
Nov 03 17:57:12.881647 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:12.879786    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7" event={"ID":"33725a51-6459-4f15-a866-e9dee41a9732","Type":"ContainerStarted","Data":"f0501a2c6ba816fc0b669d00ecfb67e0f6f224458b290b5cb9fab2df65eeb95c"}
Nov 03 17:57:12.997320 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:12.997208    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7" podStartSLOduration=1.997193883 podStartE2EDuration="1.997193883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:57:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:57:12.996633256 +0000 UTC m=+672.012831322" watchObservedRunningTime="2026-11-03 17:57:12.997193883 +0000 UTC m=+672.013391988"
Nov 03 17:57:13.250574 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:13.250036    5315 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 03 17:57:13.250574 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:13.250102    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn" podUID="cb78e2af-5c83-4d6d-a562-274b12209c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 03 17:57:13.297282 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:13.297250    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7"
Nov 03 17:57:14.250632 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:14.250150    5315 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 03 17:57:14.250632 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:14.250232    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn" podUID="cb78e2af-5c83-4d6d-a562-274b12209c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 03 17:57:15.250701 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:15.249853    5315 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 03 17:57:15.250701 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:15.249943    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn" podUID="cb78e2af-5c83-4d6d-a562-274b12209c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 03 17:57:16.251893 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:16.251856    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn"
Nov 03 17:57:16.252419 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:16.252399    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn"
Nov 03 17:57:16.253229 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:16.253205    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn"
Nov 03 17:57:18.920943 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:18.920911    5315 generic.go:356] "Generic (PLEG): container finished" podID="25a8498d-e23e-44a9-81fc-3170b124ad13" containerID="0d23428114557db8e457dc9944c3de90e0c1ea9a9527b88683aca4ec1b9a423f" exitCode=0
Nov 03 17:57:18.921385 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:18.920982    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9" event={"ID":"25a8498d-e23e-44a9-81fc-3170b124ad13","Type":"ContainerDied","Data":"0d23428114557db8e457dc9944c3de90e0c1ea9a9527b88683aca4ec1b9a423f"}
Nov 03 17:57:18.921440 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:18.921384    5315 scope.go:122] "RemoveContainer" containerID="0d23428114557db8e457dc9944c3de90e0c1ea9a9527b88683aca4ec1b9a423f"
Nov 03 17:57:18.922825 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:18.922794    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 17:57:18.922942 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:18.922882    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 17:57:18.922942 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:18.922932    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 17:57:19.941630 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:19.941178    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9" event={"ID":"25a8498d-e23e-44a9-81fc-3170b124ad13","Type":"ContainerStarted","Data":"0aefff3fa1372a9e3b95667273a96b7ba828e9bf2c79b586bd15234b49e7aacf"}
Nov 03 17:57:19.978413 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:19.978361    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Nov 03 17:57:19.978637 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:19.978437    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9" podUID="25a8498d-e23e-44a9-81fc-3170b124ad13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Nov 03 17:57:19.978760 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:19.978737    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9"
Nov 03 17:57:20.979170 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:20.979128    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Nov 03 17:57:20.979602 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:20.979201    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9" podUID="25a8498d-e23e-44a9-81fc-3170b124ad13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Nov 03 17:57:21.978872 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:21.978836    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Nov 03 17:57:21.979068 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:21.978890    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9" podUID="25a8498d-e23e-44a9-81fc-3170b124ad13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Nov 03 17:57:22.296684 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:22.296641    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7"
Nov 03 17:57:22.309042 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:22.309014    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7"
Nov 03 17:57:22.974006 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:22.973977    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7"
Nov 03 17:57:22.978434 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:22.978412    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Nov 03 17:57:22.978505 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:22.978466    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9" podUID="25a8498d-e23e-44a9-81fc-3170b124ad13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Nov 03 17:57:23.978959 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:23.978924    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Nov 03 17:57:23.979371 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:23.978980    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9" podUID="25a8498d-e23e-44a9-81fc-3170b124ad13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Nov 03 17:57:24.577068 ip-10-0-128-204 kubenswrapper[5315]: E1103 17:57:24.577031    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 17:57:24.978434 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:24.978348    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Nov 03 17:57:24.978434 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:24.978406    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9" podUID="25a8498d-e23e-44a9-81fc-3170b124ad13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Nov 03 17:57:25.983049 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:25.983020    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9"
Nov 03 17:57:25.983434 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:25.983291    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9"
Nov 03 17:57:25.984037 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:25.984021    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9"
Nov 03 17:57:44.716020 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:44.715982    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l"]
Nov 03 17:57:44.718932 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:44.718909    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l"
Nov 03 17:57:44.811689 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:44.811657    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-2b4f46\"/\"kube-root-ca.crt\""
Nov 03 17:57:44.811689 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:44.811695    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-2b4f46\"/\"openshift-service-ca.crt\""
Nov 03 17:57:44.819097 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:44.819077    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-2b4f46\"/\"default-dockercfg-hgn5n\""
Nov 03 17:57:44.819209 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:44.819138    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-2b4f46\"/\"simulator-w5-2b4f46-kserve-self-signed-certs\""
Nov 03 17:57:44.821423 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:44.821398    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l"]
Nov 03 17:57:44.897723 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:44.897684    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/114dcf54-e4eb-4294-a77b-604fe6d57d48-home\") pod \"simulator-w5-2b4f46-kserve-67884bc556-bsf5l\" (UID: \"114dcf54-e4eb-4294-a77b-604fe6d57d48\") " pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l"
Nov 03 17:57:44.897995 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:44.897974    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/114dcf54-e4eb-4294-a77b-604fe6d57d48-dshm\") pod \"simulator-w5-2b4f46-kserve-67884bc556-bsf5l\" (UID: \"114dcf54-e4eb-4294-a77b-604fe6d57d48\") " pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l"
Nov 03 17:57:44.905364 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:44.905325    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/114dcf54-e4eb-4294-a77b-604fe6d57d48-tmp-dir\") pod \"simulator-w5-2b4f46-kserve-67884bc556-bsf5l\" (UID: \"114dcf54-e4eb-4294-a77b-604fe6d57d48\") " pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l"
Nov 03 17:57:44.905631 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:44.905587    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/114dcf54-e4eb-4294-a77b-604fe6d57d48-model-cache\") pod \"simulator-w5-2b4f46-kserve-67884bc556-bsf5l\" (UID: \"114dcf54-e4eb-4294-a77b-604fe6d57d48\") " pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l"
Nov 03 17:57:44.905938 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:44.905908    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnzpl\" (UniqueName: \"kubernetes.io/projected/114dcf54-e4eb-4294-a77b-604fe6d57d48-kube-api-access-hnzpl\") pod \"simulator-w5-2b4f46-kserve-67884bc556-bsf5l\" (UID: \"114dcf54-e4eb-4294-a77b-604fe6d57d48\") " pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l"
Nov 03 17:57:44.906878 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:44.906848    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/114dcf54-e4eb-4294-a77b-604fe6d57d48-tls-certs\") pod \"simulator-w5-2b4f46-kserve-67884bc556-bsf5l\" (UID: \"114dcf54-e4eb-4294-a77b-604fe6d57d48\") " pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l"
Nov 03 17:57:45.008019 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:45.007924    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/114dcf54-e4eb-4294-a77b-604fe6d57d48-model-cache\") pod \"simulator-w5-2b4f46-kserve-67884bc556-bsf5l\" (UID: \"114dcf54-e4eb-4294-a77b-604fe6d57d48\") " pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l"
Nov 03 17:57:45.008019 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:45.007984    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnzpl\" (UniqueName: \"kubernetes.io/projected/114dcf54-e4eb-4294-a77b-604fe6d57d48-kube-api-access-hnzpl\") pod \"simulator-w5-2b4f46-kserve-67884bc556-bsf5l\" (UID: \"114dcf54-e4eb-4294-a77b-604fe6d57d48\") " pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l"
Nov 03 17:57:45.008019 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:45.008008    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/114dcf54-e4eb-4294-a77b-604fe6d57d48-tls-certs\") pod \"simulator-w5-2b4f46-kserve-67884bc556-bsf5l\" (UID: \"114dcf54-e4eb-4294-a77b-604fe6d57d48\") " pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l"
Nov 03 17:57:45.008298 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:45.008051    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/114dcf54-e4eb-4294-a77b-604fe6d57d48-home\") pod \"simulator-w5-2b4f46-kserve-67884bc556-bsf5l\" (UID: \"114dcf54-e4eb-4294-a77b-604fe6d57d48\") " pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l"
Nov 03 17:57:45.008298 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:45.008187    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/114dcf54-e4eb-4294-a77b-604fe6d57d48-dshm\") pod \"simulator-w5-2b4f46-kserve-67884bc556-bsf5l\" (UID: \"114dcf54-e4eb-4294-a77b-604fe6d57d48\") " pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l"
Nov 03 17:57:45.008402 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:45.008293    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/114dcf54-e4eb-4294-a77b-604fe6d57d48-tmp-dir\") pod \"simulator-w5-2b4f46-kserve-67884bc556-bsf5l\" (UID: \"114dcf54-e4eb-4294-a77b-604fe6d57d48\") " pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l"
Nov 03 17:57:45.008402 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:45.008363    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/114dcf54-e4eb-4294-a77b-604fe6d57d48-model-cache\") pod \"simulator-w5-2b4f46-kserve-67884bc556-bsf5l\" (UID: \"114dcf54-e4eb-4294-a77b-604fe6d57d48\") " pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l"
Nov 03 17:57:45.008476 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:45.008447    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/114dcf54-e4eb-4294-a77b-604fe6d57d48-home\") pod \"simulator-w5-2b4f46-kserve-67884bc556-bsf5l\" (UID: \"114dcf54-e4eb-4294-a77b-604fe6d57d48\") " pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l"
Nov 03 17:57:45.008707 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:45.008688    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/114dcf54-e4eb-4294-a77b-604fe6d57d48-tmp-dir\") pod \"simulator-w5-2b4f46-kserve-67884bc556-bsf5l\" (UID: \"114dcf54-e4eb-4294-a77b-604fe6d57d48\") " pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l"
Nov 03 17:57:45.010400 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:45.010382    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/114dcf54-e4eb-4294-a77b-604fe6d57d48-dshm\") pod \"simulator-w5-2b4f46-kserve-67884bc556-bsf5l\" (UID: \"114dcf54-e4eb-4294-a77b-604fe6d57d48\") " pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l"
Nov 03 17:57:45.010521 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:45.010506    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/114dcf54-e4eb-4294-a77b-604fe6d57d48-tls-certs\") pod \"simulator-w5-2b4f46-kserve-67884bc556-bsf5l\" (UID: \"114dcf54-e4eb-4294-a77b-604fe6d57d48\") " pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l"
Nov 03 17:57:45.066330 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:45.066299    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnzpl\" (UniqueName: \"kubernetes.io/projected/114dcf54-e4eb-4294-a77b-604fe6d57d48-kube-api-access-hnzpl\") pod \"simulator-w5-2b4f46-kserve-67884bc556-bsf5l\" (UID: \"114dcf54-e4eb-4294-a77b-604fe6d57d48\") " pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l"
Nov 03 17:57:45.330071 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:45.330029    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l"
Nov 03 17:57:45.698421 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:45.698335    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l"]
Nov 03 17:57:45.705064 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:57:45.705031    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod114dcf54_e4eb_4294_a77b_604fe6d57d48.slice/crio-10772cdde472eb8b664c4876fdf69760ae1ef41c8a6ecaff2b04a4068c0b264e WatchSource:0}: Error finding container 10772cdde472eb8b664c4876fdf69760ae1ef41c8a6ecaff2b04a4068c0b264e: Status 404 returned error can't find the container with id 10772cdde472eb8b664c4876fdf69760ae1ef41c8a6ecaff2b04a4068c0b264e
Nov 03 17:57:46.125647 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:46.120183    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l" event={"ID":"114dcf54-e4eb-4294-a77b-604fe6d57d48","Type":"ContainerStarted","Data":"48f26f116b804ff91ff27503181c6308ca41bba12d6db0880b978578d7d33193"}
Nov 03 17:57:46.125647 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:46.120237    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l" event={"ID":"114dcf54-e4eb-4294-a77b-604fe6d57d48","Type":"ContainerStarted","Data":"10772cdde472eb8b664c4876fdf69760ae1ef41c8a6ecaff2b04a4068c0b264e"}
Nov 03 17:57:46.246635 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:46.246283    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l" podStartSLOduration=2.246264661 podStartE2EDuration="2.246264661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:57:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:57:46.239468074 +0000 UTC m=+705.255666143" watchObservedRunningTime="2026-11-03 17:57:46.246264661 +0000 UTC m=+705.262462732"
Nov 03 17:57:46.330786 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:46.330735    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l"
Nov 03 17:57:54.577511 ip-10-0-128-204 kubenswrapper[5315]: E1103 17:57:54.577472    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 17:57:55.330191 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:55.330153    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l"
Nov 03 17:57:55.342674 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:55.342649    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l"
Nov 03 17:57:56.183148 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:57:56.183111    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l"
Nov 03 17:58:16.837439 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:16.837362    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv"]
Nov 03 17:58:16.840234 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:16.840212    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv"
Nov 03 17:58:16.900716 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:16.900684    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6d706cb-3aa5-419c-8f6c-29347bb788b1-model-cache\") pod \"premium-w6-aab753-kserve-56975cd5c9-fsrtv\" (UID: \"f6d706cb-3aa5-419c-8f6c-29347bb788b1\") " pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv"
Nov 03 17:58:16.900894 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:16.900844    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6d706cb-3aa5-419c-8f6c-29347bb788b1-tls-certs\") pod \"premium-w6-aab753-kserve-56975cd5c9-fsrtv\" (UID: \"f6d706cb-3aa5-419c-8f6c-29347bb788b1\") " pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv"
Nov 03 17:58:16.900894 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:16.900879    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqvb5\" (UniqueName: \"kubernetes.io/projected/f6d706cb-3aa5-419c-8f6c-29347bb788b1-kube-api-access-zqvb5\") pod \"premium-w6-aab753-kserve-56975cd5c9-fsrtv\" (UID: \"f6d706cb-3aa5-419c-8f6c-29347bb788b1\") " pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv"
Nov 03 17:58:16.901000 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:16.900959    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6d706cb-3aa5-419c-8f6c-29347bb788b1-home\") pod \"premium-w6-aab753-kserve-56975cd5c9-fsrtv\" (UID: \"f6d706cb-3aa5-419c-8f6c-29347bb788b1\") " pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv"
Nov 03 17:58:16.901000 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:16.900991    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6d706cb-3aa5-419c-8f6c-29347bb788b1-dshm\") pod \"premium-w6-aab753-kserve-56975cd5c9-fsrtv\" (UID: \"f6d706cb-3aa5-419c-8f6c-29347bb788b1\") " pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv"
Nov 03 17:58:16.901090 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:16.901006    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6d706cb-3aa5-419c-8f6c-29347bb788b1-tmp-dir\") pod \"premium-w6-aab753-kserve-56975cd5c9-fsrtv\" (UID: \"f6d706cb-3aa5-419c-8f6c-29347bb788b1\") " pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv"
Nov 03 17:58:16.934743 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:16.934708    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-aab753\"/\"premium-w6-aab753-kserve-self-signed-certs\""
Nov 03 17:58:16.936302 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:16.936279    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv"]
Nov 03 17:58:17.008470 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:17.005082    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6d706cb-3aa5-419c-8f6c-29347bb788b1-tls-certs\") pod \"premium-w6-aab753-kserve-56975cd5c9-fsrtv\" (UID: \"f6d706cb-3aa5-419c-8f6c-29347bb788b1\") " pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv"
Nov 03 17:58:17.008470 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:17.005135    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqvb5\" (UniqueName: \"kubernetes.io/projected/f6d706cb-3aa5-419c-8f6c-29347bb788b1-kube-api-access-zqvb5\") pod \"premium-w6-aab753-kserve-56975cd5c9-fsrtv\" (UID: \"f6d706cb-3aa5-419c-8f6c-29347bb788b1\") " pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv"
Nov 03 17:58:17.008470 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:17.005193    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6d706cb-3aa5-419c-8f6c-29347bb788b1-home\") pod \"premium-w6-aab753-kserve-56975cd5c9-fsrtv\" (UID: \"f6d706cb-3aa5-419c-8f6c-29347bb788b1\") " pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv"
Nov 03 17:58:17.008470 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:17.005235    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6d706cb-3aa5-419c-8f6c-29347bb788b1-dshm\") pod \"premium-w6-aab753-kserve-56975cd5c9-fsrtv\" (UID: \"f6d706cb-3aa5-419c-8f6c-29347bb788b1\") " pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv"
Nov 03 17:58:17.008470 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:17.005259    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6d706cb-3aa5-419c-8f6c-29347bb788b1-tmp-dir\") pod \"premium-w6-aab753-kserve-56975cd5c9-fsrtv\" (UID: \"f6d706cb-3aa5-419c-8f6c-29347bb788b1\") " pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv"
Nov 03 17:58:17.008470 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:17.005317    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6d706cb-3aa5-419c-8f6c-29347bb788b1-model-cache\") pod \"premium-w6-aab753-kserve-56975cd5c9-fsrtv\" (UID: \"f6d706cb-3aa5-419c-8f6c-29347bb788b1\") " pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv"
Nov 03 17:58:17.008470 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:17.005852    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6d706cb-3aa5-419c-8f6c-29347bb788b1-model-cache\") pod \"premium-w6-aab753-kserve-56975cd5c9-fsrtv\" (UID: \"f6d706cb-3aa5-419c-8f6c-29347bb788b1\") " pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv"
Nov 03 17:58:17.008470 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:17.006664    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6d706cb-3aa5-419c-8f6c-29347bb788b1-home\") pod \"premium-w6-aab753-kserve-56975cd5c9-fsrtv\" (UID: \"f6d706cb-3aa5-419c-8f6c-29347bb788b1\") " pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv"
Nov 03 17:58:17.008470 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:17.007522    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6d706cb-3aa5-419c-8f6c-29347bb788b1-tmp-dir\") pod \"premium-w6-aab753-kserve-56975cd5c9-fsrtv\" (UID: \"f6d706cb-3aa5-419c-8f6c-29347bb788b1\") " pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv"
Nov 03 17:58:17.013740 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:17.013693    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6d706cb-3aa5-419c-8f6c-29347bb788b1-dshm\") pod \"premium-w6-aab753-kserve-56975cd5c9-fsrtv\" (UID: \"f6d706cb-3aa5-419c-8f6c-29347bb788b1\") " pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv"
Nov 03 17:58:17.014676 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:17.014648    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6d706cb-3aa5-419c-8f6c-29347bb788b1-tls-certs\") pod \"premium-w6-aab753-kserve-56975cd5c9-fsrtv\" (UID: \"f6d706cb-3aa5-419c-8f6c-29347bb788b1\") " pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv"
Nov 03 17:58:17.070449 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:17.070416    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqvb5\" (UniqueName: \"kubernetes.io/projected/f6d706cb-3aa5-419c-8f6c-29347bb788b1-kube-api-access-zqvb5\") pod \"premium-w6-aab753-kserve-56975cd5c9-fsrtv\" (UID: \"f6d706cb-3aa5-419c-8f6c-29347bb788b1\") " pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv"
Nov 03 17:58:17.152411 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:17.152325    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv"
Nov 03 17:58:17.549627 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:17.549586    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv"]
Nov 03 17:58:17.556265 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:58:17.556233    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6d706cb_3aa5_419c_8f6c_29347bb788b1.slice/crio-4deaeda3db0ad2b7f4f35533006d02feca793df378ce6752519f321eda4c8715 WatchSource:0}: Error finding container 4deaeda3db0ad2b7f4f35533006d02feca793df378ce6752519f321eda4c8715: Status 404 returned error can't find the container with id 4deaeda3db0ad2b7f4f35533006d02feca793df378ce6752519f321eda4c8715
Nov 03 17:58:18.286331 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:18.286279    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv" event={"ID":"f6d706cb-3aa5-419c-8f6c-29347bb788b1","Type":"ContainerStarted","Data":"6b51db0b64221ebe2f34ab8d9b57bc9508616ed4b1c71fca16944f6476a6461c"}
Nov 03 17:58:18.286331 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:18.286337    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv" event={"ID":"f6d706cb-3aa5-419c-8f6c-29347bb788b1","Type":"ContainerStarted","Data":"4deaeda3db0ad2b7f4f35533006d02feca793df378ce6752519f321eda4c8715"}
Nov 03 17:58:18.397481 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:18.397428    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv" podStartSLOduration=2.397414613 podStartE2EDuration="2.397414613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:58:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:58:18.396674939 +0000 UTC m=+737.412873009" watchObservedRunningTime="2026-11-03 17:58:18.397414613 +0000 UTC m=+737.413612681"
Nov 03 17:58:19.152829 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:19.152787    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv"
Nov 03 17:58:24.577233 ip-10-0-128-204 kubenswrapper[5315]: E1103 17:58:24.577196    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 17:58:27.157633 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:27.155910    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv"
Nov 03 17:58:27.210632 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:27.210001    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv"
Nov 03 17:58:27.346306 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:27.346273    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv"
Nov 03 17:58:54.427339 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:54.427304    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr"]
Nov 03 17:58:54.430091 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:54.430059    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr"
Nov 03 17:58:54.478831 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:54.478805    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-a86228\"/\"openshift-service-ca.crt\""
Nov 03 17:58:54.487220 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:54.487185    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df8e0b71-e876-4cea-a7ae-3532c887c660-tls-certs\") pod \"premium-w4-a86228-kserve-7df468f8ff-54nfr\" (UID: \"df8e0b71-e876-4cea-a7ae-3532c887c660\") " pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr"
Nov 03 17:58:54.487340 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:54.487232    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdstr\" (UniqueName: \"kubernetes.io/projected/df8e0b71-e876-4cea-a7ae-3532c887c660-kube-api-access-xdstr\") pod \"premium-w4-a86228-kserve-7df468f8ff-54nfr\" (UID: \"df8e0b71-e876-4cea-a7ae-3532c887c660\") " pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr"
Nov 03 17:58:54.487390 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:54.487340    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df8e0b71-e876-4cea-a7ae-3532c887c660-dshm\") pod \"premium-w4-a86228-kserve-7df468f8ff-54nfr\" (UID: \"df8e0b71-e876-4cea-a7ae-3532c887c660\") " pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr"
Nov 03 17:58:54.487425 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:54.487391    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df8e0b71-e876-4cea-a7ae-3532c887c660-home\") pod \"premium-w4-a86228-kserve-7df468f8ff-54nfr\" (UID: \"df8e0b71-e876-4cea-a7ae-3532c887c660\") " pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr"
Nov 03 17:58:54.487458 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:54.487423    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df8e0b71-e876-4cea-a7ae-3532c887c660-tmp-dir\") pod \"premium-w4-a86228-kserve-7df468f8ff-54nfr\" (UID: \"df8e0b71-e876-4cea-a7ae-3532c887c660\") " pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr"
Nov 03 17:58:54.487512 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:54.487498    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df8e0b71-e876-4cea-a7ae-3532c887c660-model-cache\") pod \"premium-w4-a86228-kserve-7df468f8ff-54nfr\" (UID: \"df8e0b71-e876-4cea-a7ae-3532c887c660\") " pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr"
Nov 03 17:58:54.524790 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:54.524754    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-a86228\"/\"kube-root-ca.crt\""
Nov 03 17:58:54.524790 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:54.524779    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-a86228\"/\"default-dockercfg-dgttj\""
Nov 03 17:58:54.524983 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:54.524859    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-a86228\"/\"premium-w4-a86228-kserve-self-signed-certs\""
Nov 03 17:58:54.526546 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:54.526527    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr"]
Nov 03 17:58:54.588600 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:54.588550    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df8e0b71-e876-4cea-a7ae-3532c887c660-tmp-dir\") pod \"premium-w4-a86228-kserve-7df468f8ff-54nfr\" (UID: \"df8e0b71-e876-4cea-a7ae-3532c887c660\") " pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr"
Nov 03 17:58:54.588940 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:54.588915    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df8e0b71-e876-4cea-a7ae-3532c887c660-model-cache\") pod \"premium-w4-a86228-kserve-7df468f8ff-54nfr\" (UID: \"df8e0b71-e876-4cea-a7ae-3532c887c660\") " pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr"
Nov 03 17:58:54.589137 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:54.589100    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df8e0b71-e876-4cea-a7ae-3532c887c660-tls-certs\") pod \"premium-w4-a86228-kserve-7df468f8ff-54nfr\" (UID: \"df8e0b71-e876-4cea-a7ae-3532c887c660\") " pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr"
Nov 03 17:58:54.589251 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:54.589161    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdstr\" (UniqueName: \"kubernetes.io/projected/df8e0b71-e876-4cea-a7ae-3532c887c660-kube-api-access-xdstr\") pod \"premium-w4-a86228-kserve-7df468f8ff-54nfr\" (UID: \"df8e0b71-e876-4cea-a7ae-3532c887c660\") " pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr"
Nov 03 17:58:54.589251 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:54.589218    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df8e0b71-e876-4cea-a7ae-3532c887c660-dshm\") pod \"premium-w4-a86228-kserve-7df468f8ff-54nfr\" (UID: \"df8e0b71-e876-4cea-a7ae-3532c887c660\") " pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr"
Nov 03 17:58:54.589251 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:54.589244    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df8e0b71-e876-4cea-a7ae-3532c887c660-home\") pod \"premium-w4-a86228-kserve-7df468f8ff-54nfr\" (UID: \"df8e0b71-e876-4cea-a7ae-3532c887c660\") " pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr"
Nov 03 17:58:54.589587 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:54.589567    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df8e0b71-e876-4cea-a7ae-3532c887c660-home\") pod \"premium-w4-a86228-kserve-7df468f8ff-54nfr\" (UID: \"df8e0b71-e876-4cea-a7ae-3532c887c660\") " pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr"
Nov 03 17:58:54.589941 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:54.589850    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df8e0b71-e876-4cea-a7ae-3532c887c660-tmp-dir\") pod \"premium-w4-a86228-kserve-7df468f8ff-54nfr\" (UID: \"df8e0b71-e876-4cea-a7ae-3532c887c660\") " pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr"
Nov 03 17:58:54.592147 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:54.590584    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df8e0b71-e876-4cea-a7ae-3532c887c660-model-cache\") pod \"premium-w4-a86228-kserve-7df468f8ff-54nfr\" (UID: \"df8e0b71-e876-4cea-a7ae-3532c887c660\") " pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr"
Nov 03 17:58:54.594386 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:54.594364    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df8e0b71-e876-4cea-a7ae-3532c887c660-tls-certs\") pod \"premium-w4-a86228-kserve-7df468f8ff-54nfr\" (UID: \"df8e0b71-e876-4cea-a7ae-3532c887c660\") " pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr"
Nov 03 17:58:54.595488 ip-10-0-128-204 kubenswrapper[5315]: E1103 17:58:54.595463    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 17:58:54.596168 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:54.596148    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df8e0b71-e876-4cea-a7ae-3532c887c660-dshm\") pod \"premium-w4-a86228-kserve-7df468f8ff-54nfr\" (UID: \"df8e0b71-e876-4cea-a7ae-3532c887c660\") " pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr"
Nov 03 17:58:54.646953 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:54.646919    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdstr\" (UniqueName: \"kubernetes.io/projected/df8e0b71-e876-4cea-a7ae-3532c887c660-kube-api-access-xdstr\") pod \"premium-w4-a86228-kserve-7df468f8ff-54nfr\" (UID: \"df8e0b71-e876-4cea-a7ae-3532c887c660\") " pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr"
Nov 03 17:58:54.741215 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:54.741117    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr"
Nov 03 17:58:55.134864 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:55.134832    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr"]
Nov 03 17:58:55.141462 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:58:55.141431    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf8e0b71_e876_4cea_a7ae_3532c887c660.slice/crio-4d1322b01636dfab68fe2fc5c01456363dac6c4efbd1d012d6a8bc6b0f5ae3be WatchSource:0}: Error finding container 4d1322b01636dfab68fe2fc5c01456363dac6c4efbd1d012d6a8bc6b0f5ae3be: Status 404 returned error can't find the container with id 4d1322b01636dfab68fe2fc5c01456363dac6c4efbd1d012d6a8bc6b0f5ae3be
Nov 03 17:58:55.475676 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:55.475574    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr" event={"ID":"df8e0b71-e876-4cea-a7ae-3532c887c660","Type":"ContainerStarted","Data":"2a87abe4c0f9b40f7d2fbd81a89e003534fe58ba68921c8745660c7c03a3d6ef"}
Nov 03 17:58:55.475676 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:55.475637    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr" event={"ID":"df8e0b71-e876-4cea-a7ae-3532c887c660","Type":"ContainerStarted","Data":"4d1322b01636dfab68fe2fc5c01456363dac6c4efbd1d012d6a8bc6b0f5ae3be"}
Nov 03 17:58:55.585664 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:55.585598    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr" podStartSLOduration=1.585584217 podStartE2EDuration="1.585584217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:58:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:58:55.585110071 +0000 UTC m=+774.601308140" watchObservedRunningTime="2026-11-03 17:58:55.585584217 +0000 UTC m=+774.601782285"
Nov 03 17:58:55.741324 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:58:55.741237    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr"
Nov 03 17:59:04.741960 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:04.741921    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr"
Nov 03 17:59:04.754900 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:04.754873    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr"
Nov 03 17:59:05.541141 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:05.541106    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr"
Nov 03 17:59:24.577891 ip-10-0-128-204 kubenswrapper[5315]: E1103 17:59:24.577849    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 17:59:34.000740 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:34.000698    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q"]
Nov 03 17:59:34.003797 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:34.003772    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q"
Nov 03 17:59:34.103460 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:34.102941    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-f79c02\"/\"kube-root-ca.crt\""
Nov 03 17:59:34.103460 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:34.103060    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-f79c02\"/\"default-dockercfg-pxq8r\""
Nov 03 17:59:34.103460 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:34.103128    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-f79c02\"/\"openshift-service-ca.crt\""
Nov 03 17:59:34.107168 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:34.107134    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c974b0f1-e00b-4e4a-b193-df769068983d-tls-certs\") pod \"premium-w2-f79c02-kserve-5875b5f8b7-snz7q\" (UID: \"c974b0f1-e00b-4e4a-b193-df769068983d\") " pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q"
Nov 03 17:59:34.107378 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:34.107354    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2gnr\" (UniqueName: \"kubernetes.io/projected/c974b0f1-e00b-4e4a-b193-df769068983d-kube-api-access-v2gnr\") pod \"premium-w2-f79c02-kserve-5875b5f8b7-snz7q\" (UID: \"c974b0f1-e00b-4e4a-b193-df769068983d\") " pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q"
Nov 03 17:59:34.107560 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:34.107542    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c974b0f1-e00b-4e4a-b193-df769068983d-tmp-dir\") pod \"premium-w2-f79c02-kserve-5875b5f8b7-snz7q\" (UID: \"c974b0f1-e00b-4e4a-b193-df769068983d\") " pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q"
Nov 03 17:59:34.107768 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:34.107749    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c974b0f1-e00b-4e4a-b193-df769068983d-model-cache\") pod \"premium-w2-f79c02-kserve-5875b5f8b7-snz7q\" (UID: \"c974b0f1-e00b-4e4a-b193-df769068983d\") " pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q"
Nov 03 17:59:34.107915 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:34.107898    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c974b0f1-e00b-4e4a-b193-df769068983d-home\") pod \"premium-w2-f79c02-kserve-5875b5f8b7-snz7q\" (UID: \"c974b0f1-e00b-4e4a-b193-df769068983d\") " pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q"
Nov 03 17:59:34.108064 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:34.108045    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c974b0f1-e00b-4e4a-b193-df769068983d-dshm\") pod \"premium-w2-f79c02-kserve-5875b5f8b7-snz7q\" (UID: \"c974b0f1-e00b-4e4a-b193-df769068983d\") " pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q"
Nov 03 17:59:34.108305 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:34.108284    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-f79c02\"/\"premium-w2-f79c02-kserve-self-signed-certs\""
Nov 03 17:59:34.108434 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:34.108419    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q"]
Nov 03 17:59:34.209043 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:34.209005    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c974b0f1-e00b-4e4a-b193-df769068983d-tls-certs\") pod \"premium-w2-f79c02-kserve-5875b5f8b7-snz7q\" (UID: \"c974b0f1-e00b-4e4a-b193-df769068983d\") " pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q"
Nov 03 17:59:34.209043 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:34.209042    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2gnr\" (UniqueName: \"kubernetes.io/projected/c974b0f1-e00b-4e4a-b193-df769068983d-kube-api-access-v2gnr\") pod \"premium-w2-f79c02-kserve-5875b5f8b7-snz7q\" (UID: \"c974b0f1-e00b-4e4a-b193-df769068983d\") " pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q"
Nov 03 17:59:34.209310 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:34.209092    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c974b0f1-e00b-4e4a-b193-df769068983d-tmp-dir\") pod \"premium-w2-f79c02-kserve-5875b5f8b7-snz7q\" (UID: \"c974b0f1-e00b-4e4a-b193-df769068983d\") " pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q"
Nov 03 17:59:34.209310 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:34.209133    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c974b0f1-e00b-4e4a-b193-df769068983d-model-cache\") pod \"premium-w2-f79c02-kserve-5875b5f8b7-snz7q\" (UID: \"c974b0f1-e00b-4e4a-b193-df769068983d\") " pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q"
Nov 03 17:59:34.209310 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:34.209165    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c974b0f1-e00b-4e4a-b193-df769068983d-home\") pod \"premium-w2-f79c02-kserve-5875b5f8b7-snz7q\" (UID: \"c974b0f1-e00b-4e4a-b193-df769068983d\") " pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q"
Nov 03 17:59:34.209310 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:34.209210    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c974b0f1-e00b-4e4a-b193-df769068983d-dshm\") pod \"premium-w2-f79c02-kserve-5875b5f8b7-snz7q\" (UID: \"c974b0f1-e00b-4e4a-b193-df769068983d\") " pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q"
Nov 03 17:59:34.209573 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:34.209551    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c974b0f1-e00b-4e4a-b193-df769068983d-tmp-dir\") pod \"premium-w2-f79c02-kserve-5875b5f8b7-snz7q\" (UID: \"c974b0f1-e00b-4e4a-b193-df769068983d\") " pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q"
Nov 03 17:59:34.209690 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:34.209577    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c974b0f1-e00b-4e4a-b193-df769068983d-home\") pod \"premium-w2-f79c02-kserve-5875b5f8b7-snz7q\" (UID: \"c974b0f1-e00b-4e4a-b193-df769068983d\") " pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q"
Nov 03 17:59:34.209690 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:34.209646    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c974b0f1-e00b-4e4a-b193-df769068983d-model-cache\") pod \"premium-w2-f79c02-kserve-5875b5f8b7-snz7q\" (UID: \"c974b0f1-e00b-4e4a-b193-df769068983d\") " pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q"
Nov 03 17:59:34.211542 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:34.211514    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c974b0f1-e00b-4e4a-b193-df769068983d-dshm\") pod \"premium-w2-f79c02-kserve-5875b5f8b7-snz7q\" (UID: \"c974b0f1-e00b-4e4a-b193-df769068983d\") " pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q"
Nov 03 17:59:34.211836 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:34.211815    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c974b0f1-e00b-4e4a-b193-df769068983d-tls-certs\") pod \"premium-w2-f79c02-kserve-5875b5f8b7-snz7q\" (UID: \"c974b0f1-e00b-4e4a-b193-df769068983d\") " pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q"
Nov 03 17:59:34.284227 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:34.284190    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2gnr\" (UniqueName: \"kubernetes.io/projected/c974b0f1-e00b-4e4a-b193-df769068983d-kube-api-access-v2gnr\") pod \"premium-w2-f79c02-kserve-5875b5f8b7-snz7q\" (UID: \"c974b0f1-e00b-4e4a-b193-df769068983d\") " pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q"
Nov 03 17:59:34.314031 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:34.313996    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q"
Nov 03 17:59:34.727992 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:34.727901    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q"]
Nov 03 17:59:34.739182 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:59:34.739151    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc974b0f1_e00b_4e4a_b193_df769068983d.slice/crio-31320146e8a2b9b695ed96ab2a1677fb6f1563408177578ddf2627b777af786f WatchSource:0}: Error finding container 31320146e8a2b9b695ed96ab2a1677fb6f1563408177578ddf2627b777af786f: Status 404 returned error can't find the container with id 31320146e8a2b9b695ed96ab2a1677fb6f1563408177578ddf2627b777af786f
Nov 03 17:59:35.679444 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:35.679406    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q" event={"ID":"c974b0f1-e00b-4e4a-b193-df769068983d","Type":"ContainerStarted","Data":"e7574d3f726d08af068b3943873d9a4c520307ba566a7ac1acf20e5296e866a0"}
Nov 03 17:59:35.679444 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:35.679449    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q" event={"ID":"c974b0f1-e00b-4e4a-b193-df769068983d","Type":"ContainerStarted","Data":"31320146e8a2b9b695ed96ab2a1677fb6f1563408177578ddf2627b777af786f"}
Nov 03 17:59:35.794213 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:35.794147    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q" podStartSLOduration=2.79412919 podStartE2EDuration="2.79412919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:59:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:59:35.792750553 +0000 UTC m=+814.808948622" watchObservedRunningTime="2026-11-03 17:59:35.79412919 +0000 UTC m=+814.810327261"
Nov 03 17:59:36.314924 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:36.314882    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q"
Nov 03 17:59:41.403976 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:41.403946    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d6c743e-jfx4h"]
Nov 03 17:59:41.406992 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:41.406970    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6c743e-jfx4h"
Nov 03 17:59:41.458544 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:41.458513    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:59:41.493703 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:41.493670    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40bf5d8c-f779-48b5-8273-e7ae30a98537-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d6c743e-jfx4h\" (UID: \"40bf5d8c-f779-48b5-8273-e7ae30a98537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6c743e-jfx4h"
Nov 03 17:59:41.493844 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:41.493769    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40bf5d8c-f779-48b5-8273-e7ae30a98537-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d6c743e-jfx4h\" (UID: \"40bf5d8c-f779-48b5-8273-e7ae30a98537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6c743e-jfx4h"
Nov 03 17:59:41.493950 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:41.493925    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhwmf\" (UniqueName: \"kubernetes.io/projected/40bf5d8c-f779-48b5-8273-e7ae30a98537-kube-api-access-xhwmf\") pod \"maas-api-revoke-sub-9d6c743e-jfx4h\" (UID: \"40bf5d8c-f779-48b5-8273-e7ae30a98537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6c743e-jfx4h"
Nov 03 17:59:41.503581 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:41.503553    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:59:41.503720 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:41.503585    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:59:41.503720 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:41.503680    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 17:59:41.506473 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:41.506451    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d6c743e-jfx4h"]
Nov 03 17:59:41.594692 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:41.594658    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40bf5d8c-f779-48b5-8273-e7ae30a98537-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d6c743e-jfx4h\" (UID: \"40bf5d8c-f779-48b5-8273-e7ae30a98537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6c743e-jfx4h"
Nov 03 17:59:41.594910 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:41.594721    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhwmf\" (UniqueName: \"kubernetes.io/projected/40bf5d8c-f779-48b5-8273-e7ae30a98537-kube-api-access-xhwmf\") pod \"maas-api-revoke-sub-9d6c743e-jfx4h\" (UID: \"40bf5d8c-f779-48b5-8273-e7ae30a98537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6c743e-jfx4h"
Nov 03 17:59:41.594910 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:41.594894    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40bf5d8c-f779-48b5-8273-e7ae30a98537-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d6c743e-jfx4h\" (UID: \"40bf5d8c-f779-48b5-8273-e7ae30a98537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6c743e-jfx4h"
Nov 03 17:59:41.595320 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:41.595295    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40bf5d8c-f779-48b5-8273-e7ae30a98537-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d6c743e-jfx4h\" (UID: \"40bf5d8c-f779-48b5-8273-e7ae30a98537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6c743e-jfx4h"
Nov 03 17:59:41.597274 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:41.597249    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40bf5d8c-f779-48b5-8273-e7ae30a98537-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d6c743e-jfx4h\" (UID: \"40bf5d8c-f779-48b5-8273-e7ae30a98537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6c743e-jfx4h"
Nov 03 17:59:41.655686 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:41.654265    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhwmf\" (UniqueName: \"kubernetes.io/projected/40bf5d8c-f779-48b5-8273-e7ae30a98537-kube-api-access-xhwmf\") pod \"maas-api-revoke-sub-9d6c743e-jfx4h\" (UID: \"40bf5d8c-f779-48b5-8273-e7ae30a98537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6c743e-jfx4h"
Nov 03 17:59:41.749789 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:41.749755    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6c743e-jfx4h"
Nov 03 17:59:42.128574 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:42.128536    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d6c743e-jfx4h"]
Nov 03 17:59:42.136814 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:59:42.136787    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40bf5d8c_f779_48b5_8273_e7ae30a98537.slice/crio-d3b78424df1b22b7143f218d4e89b2a1a2945da671a4b4385661f7959c7000ff WatchSource:0}: Error finding container d3b78424df1b22b7143f218d4e89b2a1a2945da671a4b4385661f7959c7000ff: Status 404 returned error can't find the container with id d3b78424df1b22b7143f218d4e89b2a1a2945da671a4b4385661f7959c7000ff
Nov 03 17:59:42.139286 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:42.139261    5315 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 17:59:42.718457 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:42.718419    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6c743e-jfx4h" event={"ID":"40bf5d8c-f779-48b5-8273-e7ae30a98537","Type":"ContainerStarted","Data":"d3b78424df1b22b7143f218d4e89b2a1a2945da671a4b4385661f7959c7000ff"}
Nov 03 17:59:44.314762 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:44.314717    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q"
Nov 03 17:59:44.332808 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:44.332780    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q"
Nov 03 17:59:44.742046 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:44.741954    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q"
Nov 03 17:59:47.751697 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:47.751665    5315 generic.go:356] "Generic (PLEG): container finished" podID="40bf5d8c-f779-48b5-8273-e7ae30a98537" containerID="fad7732a6044a3f44384900b623768236ddbf2c33b3d16715c39bc3155307870" exitCode=0
Nov 03 17:59:47.752111 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:47.751730    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6c743e-jfx4h" event={"ID":"40bf5d8c-f779-48b5-8273-e7ae30a98537","Type":"ContainerDied","Data":"fad7732a6044a3f44384900b623768236ddbf2c33b3d16715c39bc3155307870"}
Nov 03 17:59:48.893947 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:48.893921    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6c743e-jfx4h"
Nov 03 17:59:49.001442 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:49.001409    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40bf5d8c-f779-48b5-8273-e7ae30a98537-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40bf5d8c-f779-48b5-8273-e7ae30a98537-maas-api-openshift-ca\") pod \"40bf5d8c-f779-48b5-8273-e7ae30a98537\" (UID: \"40bf5d8c-f779-48b5-8273-e7ae30a98537\") "
Nov 03 17:59:49.001605 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:49.001457    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40bf5d8c-f779-48b5-8273-e7ae30a98537-kube-api-access-xhwmf\" (UniqueName: \"kubernetes.io/projected/40bf5d8c-f779-48b5-8273-e7ae30a98537-kube-api-access-xhwmf\") pod \"40bf5d8c-f779-48b5-8273-e7ae30a98537\" (UID: \"40bf5d8c-f779-48b5-8273-e7ae30a98537\") "
Nov 03 17:59:49.001605 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:49.001549    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40bf5d8c-f779-48b5-8273-e7ae30a98537-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40bf5d8c-f779-48b5-8273-e7ae30a98537-maas-api-xks-ca\") pod \"40bf5d8c-f779-48b5-8273-e7ae30a98537\" (UID: \"40bf5d8c-f779-48b5-8273-e7ae30a98537\") "
Nov 03 17:59:49.001823 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:49.001799    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40bf5d8c-f779-48b5-8273-e7ae30a98537-maas-api-openshift-ca" pod "40bf5d8c-f779-48b5-8273-e7ae30a98537" (UID: "40bf5d8c-f779-48b5-8273-e7ae30a98537"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:59:49.001990 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:49.001967    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40bf5d8c-f779-48b5-8273-e7ae30a98537-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:59:49.003744 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:49.003721    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40bf5d8c-f779-48b5-8273-e7ae30a98537-kube-api-access-xhwmf" pod "40bf5d8c-f779-48b5-8273-e7ae30a98537" (UID: "40bf5d8c-f779-48b5-8273-e7ae30a98537"). InnerVolumeSpecName "kube-api-access-xhwmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:59:49.003744 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:49.003729    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40bf5d8c-f779-48b5-8273-e7ae30a98537-maas-api-xks-ca" pod "40bf5d8c-f779-48b5-8273-e7ae30a98537" (UID: "40bf5d8c-f779-48b5-8273-e7ae30a98537"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:59:49.102525 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:49.102492    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40bf5d8c-f779-48b5-8273-e7ae30a98537-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:59:49.102525 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:49.102523    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhwmf\" (UniqueName: \"kubernetes.io/projected/40bf5d8c-f779-48b5-8273-e7ae30a98537-kube-api-access-xhwmf\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:59:49.763080 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:49.763043    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6c743e-jfx4h"
Nov 03 17:59:49.763080 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:49.763065    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6c743e-jfx4h" event={"ID":"40bf5d8c-f779-48b5-8273-e7ae30a98537","Type":"ContainerDied","Data":"d3b78424df1b22b7143f218d4e89b2a1a2945da671a4b4385661f7959c7000ff"}
Nov 03 17:59:49.763297 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:49.763118    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3b78424df1b22b7143f218d4e89b2a1a2945da671a4b4385661f7959c7000ff"
Nov 03 17:59:51.788333 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:51.788299    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85b3e2e2-z982z"]
Nov 03 17:59:51.788817 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:51.788752    5315 cpu_manager.go:414] "containerMap: removing container" podUID="40bf5d8c-f779-48b5-8273-e7ae30a98537" containerName="revoke-keys"
Nov 03 17:59:51.788817 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:51.788764    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40bf5d8c-f779-48b5-8273-e7ae30a98537" containerName="revoke-keys"
Nov 03 17:59:51.788886 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:51.788850    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="40bf5d8c-f779-48b5-8273-e7ae30a98537" containerName="revoke-keys"
Nov 03 17:59:51.792023 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:51.791998    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b3e2e2-z982z"
Nov 03 17:59:51.826630 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:51.826563    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/402fd518-42cc-4242-8e14-96e762de3f13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85b3e2e2-z982z\" (UID: \"402fd518-42cc-4242-8e14-96e762de3f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b3e2e2-z982z"
Nov 03 17:59:51.826630 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:51.826621    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx4s6\" (UniqueName: \"kubernetes.io/projected/402fd518-42cc-4242-8e14-96e762de3f13-kube-api-access-nx4s6\") pod \"maas-api-revoke-sub-85b3e2e2-z982z\" (UID: \"402fd518-42cc-4242-8e14-96e762de3f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b3e2e2-z982z"
Nov 03 17:59:51.826864 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:51.826728    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/402fd518-42cc-4242-8e14-96e762de3f13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85b3e2e2-z982z\" (UID: \"402fd518-42cc-4242-8e14-96e762de3f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b3e2e2-z982z"
Nov 03 17:59:51.841500 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:51.841471    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:59:51.887490 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:51.887451    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:59:51.887720 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:51.887555    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 17:59:51.887808 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:51.887791    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:59:51.892737 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:51.892712    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85b3e2e2-z982z"]
Nov 03 17:59:51.927854 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:51.927817    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/402fd518-42cc-4242-8e14-96e762de3f13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85b3e2e2-z982z\" (UID: \"402fd518-42cc-4242-8e14-96e762de3f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b3e2e2-z982z"
Nov 03 17:59:51.928051 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:51.927862    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx4s6\" (UniqueName: \"kubernetes.io/projected/402fd518-42cc-4242-8e14-96e762de3f13-kube-api-access-nx4s6\") pod \"maas-api-revoke-sub-85b3e2e2-z982z\" (UID: \"402fd518-42cc-4242-8e14-96e762de3f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b3e2e2-z982z"
Nov 03 17:59:51.928051 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:51.927911    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/402fd518-42cc-4242-8e14-96e762de3f13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85b3e2e2-z982z\" (UID: \"402fd518-42cc-4242-8e14-96e762de3f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b3e2e2-z982z"
Nov 03 17:59:51.928558 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:51.928537    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/402fd518-42cc-4242-8e14-96e762de3f13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85b3e2e2-z982z\" (UID: \"402fd518-42cc-4242-8e14-96e762de3f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b3e2e2-z982z"
Nov 03 17:59:51.930211 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:51.930186    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/402fd518-42cc-4242-8e14-96e762de3f13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85b3e2e2-z982z\" (UID: \"402fd518-42cc-4242-8e14-96e762de3f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b3e2e2-z982z"
Nov 03 17:59:51.986504 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:51.986473    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx4s6\" (UniqueName: \"kubernetes.io/projected/402fd518-42cc-4242-8e14-96e762de3f13-kube-api-access-nx4s6\") pod \"maas-api-revoke-sub-85b3e2e2-z982z\" (UID: \"402fd518-42cc-4242-8e14-96e762de3f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b3e2e2-z982z"
Nov 03 17:59:52.105308 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:52.105222    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b3e2e2-z982z"
Nov 03 17:59:52.475662 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:52.475566    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85b3e2e2-z982z"]
Nov 03 17:59:52.482192 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:59:52.482155    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod402fd518_42cc_4242_8e14_96e762de3f13.slice/crio-4e1ab5955f9e7d4e41a4c81bd14447615099f634f5cd48fe1d6c37ba716444cd WatchSource:0}: Error finding container 4e1ab5955f9e7d4e41a4c81bd14447615099f634f5cd48fe1d6c37ba716444cd: Status 404 returned error can't find the container with id 4e1ab5955f9e7d4e41a4c81bd14447615099f634f5cd48fe1d6c37ba716444cd
Nov 03 17:59:52.780024 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:52.779990    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b3e2e2-z982z" event={"ID":"402fd518-42cc-4242-8e14-96e762de3f13","Type":"ContainerStarted","Data":"c7f990e9ea27cdf2f40787b6896ec7ed42dee1b36ddfa7f836333050ca3075f9"}
Nov 03 17:59:52.780187 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:52.780052    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b3e2e2-z982z" event={"ID":"402fd518-42cc-4242-8e14-96e762de3f13","Type":"ContainerStarted","Data":"4e1ab5955f9e7d4e41a4c81bd14447615099f634f5cd48fe1d6c37ba716444cd"}
Nov 03 17:59:52.889811 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:52.889751    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b3e2e2-z982z" podStartSLOduration=1.8897308819999998 podStartE2EDuration="1.889730882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:59:52.888723676 +0000 UTC m=+831.904921953" watchObservedRunningTime="2026-11-03 17:59:52.889730882 +0000 UTC m=+831.905928954"
Nov 03 17:59:53.786436 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:53.786398    5315 generic.go:356] "Generic (PLEG): container finished" podID="402fd518-42cc-4242-8e14-96e762de3f13" containerID="c7f990e9ea27cdf2f40787b6896ec7ed42dee1b36ddfa7f836333050ca3075f9" exitCode=0
Nov 03 17:59:53.786660 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:53.786484    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b3e2e2-z982z" event={"ID":"402fd518-42cc-4242-8e14-96e762de3f13","Type":"ContainerDied","Data":"c7f990e9ea27cdf2f40787b6896ec7ed42dee1b36ddfa7f836333050ca3075f9"}
Nov 03 17:59:54.577224 ip-10-0-128-204 kubenswrapper[5315]: E1103 17:59:54.577182    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 17:59:54.936791 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:54.936764    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b3e2e2-z982z"
Nov 03 17:59:54.954267 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:54.954235    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/402fd518-42cc-4242-8e14-96e762de3f13-kube-api-access-nx4s6\" (UniqueName: \"kubernetes.io/projected/402fd518-42cc-4242-8e14-96e762de3f13-kube-api-access-nx4s6\") pod \"402fd518-42cc-4242-8e14-96e762de3f13\" (UID: \"402fd518-42cc-4242-8e14-96e762de3f13\") "
Nov 03 17:59:54.954423 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:54.954291    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/402fd518-42cc-4242-8e14-96e762de3f13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/402fd518-42cc-4242-8e14-96e762de3f13-maas-api-openshift-ca\") pod \"402fd518-42cc-4242-8e14-96e762de3f13\" (UID: \"402fd518-42cc-4242-8e14-96e762de3f13\") "
Nov 03 17:59:54.954423 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:54.954359    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/402fd518-42cc-4242-8e14-96e762de3f13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/402fd518-42cc-4242-8e14-96e762de3f13-maas-api-xks-ca\") pod \"402fd518-42cc-4242-8e14-96e762de3f13\" (UID: \"402fd518-42cc-4242-8e14-96e762de3f13\") "
Nov 03 17:59:54.954675 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:54.954651    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/402fd518-42cc-4242-8e14-96e762de3f13-maas-api-openshift-ca" pod "402fd518-42cc-4242-8e14-96e762de3f13" (UID: "402fd518-42cc-4242-8e14-96e762de3f13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:59:54.956347 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:54.956318    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/402fd518-42cc-4242-8e14-96e762de3f13-maas-api-xks-ca" pod "402fd518-42cc-4242-8e14-96e762de3f13" (UID: "402fd518-42cc-4242-8e14-96e762de3f13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:59:54.956347 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:54.956339    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/402fd518-42cc-4242-8e14-96e762de3f13-kube-api-access-nx4s6" pod "402fd518-42cc-4242-8e14-96e762de3f13" (UID: "402fd518-42cc-4242-8e14-96e762de3f13"). InnerVolumeSpecName "kube-api-access-nx4s6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:59:55.055331 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:55.055297    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/402fd518-42cc-4242-8e14-96e762de3f13-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:59:55.055331 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:55.055329    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx4s6\" (UniqueName: \"kubernetes.io/projected/402fd518-42cc-4242-8e14-96e762de3f13-kube-api-access-nx4s6\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:59:55.055527 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:55.055343    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/402fd518-42cc-4242-8e14-96e762de3f13-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:59:55.798626 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:55.798586    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b3e2e2-z982z"
Nov 03 17:59:55.799058 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:55.798587    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b3e2e2-z982z" event={"ID":"402fd518-42cc-4242-8e14-96e762de3f13","Type":"ContainerDied","Data":"4e1ab5955f9e7d4e41a4c81bd14447615099f634f5cd48fe1d6c37ba716444cd"}
Nov 03 17:59:55.799058 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:55.798744    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e1ab5955f9e7d4e41a4c81bd14447615099f634f5cd48fe1d6c37ba716444cd"
Nov 03 17:59:57.128237 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:57.128199    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d3ab78f-bvrnk"]
Nov 03 17:59:57.128879 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:57.128856    5315 cpu_manager.go:414] "containerMap: removing container" podUID="402fd518-42cc-4242-8e14-96e762de3f13" containerName="revoke-keys"
Nov 03 17:59:57.128949 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:57.128882    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="402fd518-42cc-4242-8e14-96e762de3f13" containerName="revoke-keys"
Nov 03 17:59:57.129024 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:57.129008    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="402fd518-42cc-4242-8e14-96e762de3f13" containerName="revoke-keys"
Nov 03 17:59:57.134048 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:57.134024    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d3ab78f-bvrnk"
Nov 03 17:59:57.176745 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:57.176705    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf859d09-eac4-4450-9269-e8cb2c3020f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d3ab78f-bvrnk\" (UID: \"cf859d09-eac4-4450-9269-e8cb2c3020f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d3ab78f-bvrnk"
Nov 03 17:59:57.177168 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:57.177134    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csjrd\" (UniqueName: \"kubernetes.io/projected/cf859d09-eac4-4450-9269-e8cb2c3020f5-kube-api-access-csjrd\") pod \"maas-api-revoke-sub-9d3ab78f-bvrnk\" (UID: \"cf859d09-eac4-4450-9269-e8cb2c3020f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d3ab78f-bvrnk"
Nov 03 17:59:57.177360 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:57.177204    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf859d09-eac4-4450-9269-e8cb2c3020f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d3ab78f-bvrnk\" (UID: \"cf859d09-eac4-4450-9269-e8cb2c3020f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d3ab78f-bvrnk"
Nov 03 17:59:57.179096 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:57.179072    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d3ab78f-bvrnk"]
Nov 03 17:59:57.189880 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:57.189844    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:59:57.190003 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:57.189935    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:59:57.223003 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:57.222975    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:59:57.223003 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:57.222988    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 17:59:57.296286 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:57.289933    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csjrd\" (UniqueName: \"kubernetes.io/projected/cf859d09-eac4-4450-9269-e8cb2c3020f5-kube-api-access-csjrd\") pod \"maas-api-revoke-sub-9d3ab78f-bvrnk\" (UID: \"cf859d09-eac4-4450-9269-e8cb2c3020f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d3ab78f-bvrnk"
Nov 03 17:59:57.296286 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:57.290008    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf859d09-eac4-4450-9269-e8cb2c3020f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d3ab78f-bvrnk\" (UID: \"cf859d09-eac4-4450-9269-e8cb2c3020f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d3ab78f-bvrnk"
Nov 03 17:59:57.296286 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:57.290110    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf859d09-eac4-4450-9269-e8cb2c3020f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d3ab78f-bvrnk\" (UID: \"cf859d09-eac4-4450-9269-e8cb2c3020f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d3ab78f-bvrnk"
Nov 03 17:59:57.296286 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:57.291013    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf859d09-eac4-4450-9269-e8cb2c3020f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d3ab78f-bvrnk\" (UID: \"cf859d09-eac4-4450-9269-e8cb2c3020f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d3ab78f-bvrnk"
Nov 03 17:59:57.309647 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:57.309221    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf859d09-eac4-4450-9269-e8cb2c3020f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d3ab78f-bvrnk\" (UID: \"cf859d09-eac4-4450-9269-e8cb2c3020f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d3ab78f-bvrnk"
Nov 03 17:59:57.349217 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:57.349184    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csjrd\" (UniqueName: \"kubernetes.io/projected/cf859d09-eac4-4450-9269-e8cb2c3020f5-kube-api-access-csjrd\") pod \"maas-api-revoke-sub-9d3ab78f-bvrnk\" (UID: \"cf859d09-eac4-4450-9269-e8cb2c3020f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d3ab78f-bvrnk"
Nov 03 17:59:57.452091 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:57.452002    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d3ab78f-bvrnk"
Nov 03 17:59:57.844979 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:57.844947    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d3ab78f-bvrnk"]
Nov 03 17:59:57.851690 ip-10-0-128-204 kubenswrapper[5315]: W1103 17:59:57.851662    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf859d09_eac4_4450_9269_e8cb2c3020f5.slice/crio-b8ede2b342bd52d2e8816d6dbeba137e6679a4d500ddc1519b304326346e1a91 WatchSource:0}: Error finding container b8ede2b342bd52d2e8816d6dbeba137e6679a4d500ddc1519b304326346e1a91: Status 404 returned error can't find the container with id b8ede2b342bd52d2e8816d6dbeba137e6679a4d500ddc1519b304326346e1a91
Nov 03 17:59:58.816244 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:58.816209    5315 generic.go:356] "Generic (PLEG): container finished" podID="cf859d09-eac4-4450-9269-e8cb2c3020f5" containerID="4d61ad13581a65fa19a21783fc9d56b898e66afe353e6cdb35d4f95ee3ffec83" exitCode=0
Nov 03 17:59:58.816714 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:58.816289    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d3ab78f-bvrnk" event={"ID":"cf859d09-eac4-4450-9269-e8cb2c3020f5","Type":"ContainerDied","Data":"4d61ad13581a65fa19a21783fc9d56b898e66afe353e6cdb35d4f95ee3ffec83"}
Nov 03 17:59:58.816714 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:58.816346    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d3ab78f-bvrnk" event={"ID":"cf859d09-eac4-4450-9269-e8cb2c3020f5","Type":"ContainerStarted","Data":"b8ede2b342bd52d2e8816d6dbeba137e6679a4d500ddc1519b304326346e1a91"}
Nov 03 17:59:59.960231 ip-10-0-128-204 kubenswrapper[5315]: I1103 17:59:59.960199    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d3ab78f-bvrnk"
Nov 03 18:00:00.024804 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.022979    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf859d09-eac4-4450-9269-e8cb2c3020f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf859d09-eac4-4450-9269-e8cb2c3020f5-maas-api-openshift-ca\") pod \"cf859d09-eac4-4450-9269-e8cb2c3020f5\" (UID: \"cf859d09-eac4-4450-9269-e8cb2c3020f5\") "
Nov 03 18:00:00.024804 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.023203    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf859d09-eac4-4450-9269-e8cb2c3020f5-kube-api-access-csjrd\" (UniqueName: \"kubernetes.io/projected/cf859d09-eac4-4450-9269-e8cb2c3020f5-kube-api-access-csjrd\") pod \"cf859d09-eac4-4450-9269-e8cb2c3020f5\" (UID: \"cf859d09-eac4-4450-9269-e8cb2c3020f5\") "
Nov 03 18:00:00.024804 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.023248    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf859d09-eac4-4450-9269-e8cb2c3020f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf859d09-eac4-4450-9269-e8cb2c3020f5-maas-api-xks-ca\") pod \"cf859d09-eac4-4450-9269-e8cb2c3020f5\" (UID: \"cf859d09-eac4-4450-9269-e8cb2c3020f5\") "
Nov 03 18:00:00.027512 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.027476    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf859d09-eac4-4450-9269-e8cb2c3020f5-maas-api-openshift-ca" pod "cf859d09-eac4-4450-9269-e8cb2c3020f5" (UID: "cf859d09-eac4-4450-9269-e8cb2c3020f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:00:00.030806 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.030758    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf859d09-eac4-4450-9269-e8cb2c3020f5-kube-api-access-csjrd" pod "cf859d09-eac4-4450-9269-e8cb2c3020f5" (UID: "cf859d09-eac4-4450-9269-e8cb2c3020f5"). InnerVolumeSpecName "kube-api-access-csjrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:00:00.032418 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.032389    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf859d09-eac4-4450-9269-e8cb2c3020f5-maas-api-xks-ca" pod "cf859d09-eac4-4450-9269-e8cb2c3020f5" (UID: "cf859d09-eac4-4450-9269-e8cb2c3020f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:00:00.124704 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.124595    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf859d09-eac4-4450-9269-e8cb2c3020f5-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:00:00.124704 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.124650    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csjrd\" (UniqueName: \"kubernetes.io/projected/cf859d09-eac4-4450-9269-e8cb2c3020f5-kube-api-access-csjrd\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:00:00.124704 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.124661    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf859d09-eac4-4450-9269-e8cb2c3020f5-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:00:00.171590 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.171550    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895480-xznh8"]
Nov 03 18:00:00.172021 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.172007    5315 cpu_manager.go:414] "containerMap: removing container" podUID="cf859d09-eac4-4450-9269-e8cb2c3020f5" containerName="revoke-keys"
Nov 03 18:00:00.172066 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.172022    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf859d09-eac4-4450-9269-e8cb2c3020f5" containerName="revoke-keys"
Nov 03 18:00:00.172120 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.172110    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf859d09-eac4-4450-9269-e8cb2c3020f5" containerName="revoke-keys"
Nov 03 18:00:00.175656 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.175627    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895480-xznh8"
Nov 03 18:00:00.176044 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.176002    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895480-zb2db"]
Nov 03 18:00:00.180126 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.180108    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895480-sg5b6"]
Nov 03 18:00:00.180266 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.180242    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895480-zb2db"
Nov 03 18:00:00.183495 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.183472    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-sg5b6"
Nov 03 18:00:00.198209 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.198182    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0ccbb9-29895480-wg82z"]
Nov 03 18:00:00.202225 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.202202    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895480-v2xwh"]
Nov 03 18:00:00.202335 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.202315    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0ccbb9-29895480-wg82z"
Nov 03 18:00:00.205734 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.205717    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895480-v2xwh"
Nov 03 18:00:00.226127 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.226088    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn5wx\" (UniqueName: \"kubernetes.io/projected/a8c7d4f4-916e-4418-aa0a-2e7e844583ba-kube-api-access-gn5wx\") pod \"maas-api-key-cleanup-29895480-sg5b6\" (UID: \"a8c7d4f4-916e-4418-aa0a-2e7e844583ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-sg5b6"
Nov 03 18:00:00.226240 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.226222    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8c7d4f4-916e-4418-aa0a-2e7e844583ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895480-sg5b6\" (UID: \"a8c7d4f4-916e-4418-aa0a-2e7e844583ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-sg5b6"
Nov 03 18:00:00.226284 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.226252    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/944b5b5f-42cf-4c48-8676-b36b0b4dd49f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-aab753-29895480-xznh8\" (UID: \"944b5b5f-42cf-4c48-8676-b36b0b4dd49f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895480-xznh8"
Nov 03 18:00:00.226351 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.226337    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkqkq\" (UniqueName: \"kubernetes.io/projected/5026ac7e-119a-4c3d-865c-848d68f35bdf-kube-api-access-hkqkq\") pod \"maas-api-key-cleanup-e2e-worker-w4-a86228-29895480-zb2db\" (UID: \"5026ac7e-119a-4c3d-865c-848d68f35bdf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895480-zb2db"
Nov 03 18:00:00.226411 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.226396    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kwwc\" (UniqueName: \"kubernetes.io/projected/944b5b5f-42cf-4c48-8676-b36b0b4dd49f-kube-api-access-8kwwc\") pod \"maas-api-key-cleanup-e2e-worker-w6-aab753-29895480-xznh8\" (UID: \"944b5b5f-42cf-4c48-8676-b36b0b4dd49f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895480-xznh8"
Nov 03 18:00:00.226457 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.226431    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5026ac7e-119a-4c3d-865c-848d68f35bdf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-a86228-29895480-zb2db\" (UID: \"5026ac7e-119a-4c3d-865c-848d68f35bdf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895480-zb2db"
Nov 03 18:00:00.237920 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.237895    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895480-xznh8"]
Nov 03 18:00:00.282374 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.282343    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895480-zb2db"]
Nov 03 18:00:00.288561 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.288528    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895480-sg5b6"]
Nov 03 18:00:00.293813 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.293785    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895480-v2xwh"]
Nov 03 18:00:00.305728 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.305698    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0ccbb9-29895480-wg82z"]
Nov 03 18:00:00.327482 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.327447    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8c7d4f4-916e-4418-aa0a-2e7e844583ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895480-sg5b6\" (UID: \"a8c7d4f4-916e-4418-aa0a-2e7e844583ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-sg5b6"
Nov 03 18:00:00.327713 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.327496    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/944b5b5f-42cf-4c48-8676-b36b0b4dd49f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-aab753-29895480-xznh8\" (UID: \"944b5b5f-42cf-4c48-8676-b36b0b4dd49f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895480-xznh8"
Nov 03 18:00:00.327713 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.327567    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkqkq\" (UniqueName: \"kubernetes.io/projected/5026ac7e-119a-4c3d-865c-848d68f35bdf-kube-api-access-hkqkq\") pod \"maas-api-key-cleanup-e2e-worker-w4-a86228-29895480-zb2db\" (UID: \"5026ac7e-119a-4c3d-865c-848d68f35bdf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895480-zb2db"
Nov 03 18:00:00.327713 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.327596    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kwwc\" (UniqueName: \"kubernetes.io/projected/944b5b5f-42cf-4c48-8676-b36b0b4dd49f-kube-api-access-8kwwc\") pod \"maas-api-key-cleanup-e2e-worker-w6-aab753-29895480-xznh8\" (UID: \"944b5b5f-42cf-4c48-8676-b36b0b4dd49f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895480-xznh8"
Nov 03 18:00:00.327713 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.327651    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5026ac7e-119a-4c3d-865c-848d68f35bdf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-a86228-29895480-zb2db\" (UID: \"5026ac7e-119a-4c3d-865c-848d68f35bdf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895480-zb2db"
Nov 03 18:00:00.327713 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.327697    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfmsg\" (UniqueName: \"kubernetes.io/projected/870683a9-9f9e-4071-a4e2-4e62a0178309-kube-api-access-xfmsg\") pod \"maas-api-key-cleanup-e2e-worker-w1-ef9633-29895480-v2xwh\" (UID: \"870683a9-9f9e-4071-a4e2-4e62a0178309\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895480-v2xwh"
Nov 03 18:00:00.327955 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.327734    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/870683a9-9f9e-4071-a4e2-4e62a0178309-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-ef9633-29895480-v2xwh\" (UID: \"870683a9-9f9e-4071-a4e2-4e62a0178309\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895480-v2xwh"
Nov 03 18:00:00.327955 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.327865    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn5wx\" (UniqueName: \"kubernetes.io/projected/a8c7d4f4-916e-4418-aa0a-2e7e844583ba-kube-api-access-gn5wx\") pod \"maas-api-key-cleanup-29895480-sg5b6\" (UID: \"a8c7d4f4-916e-4418-aa0a-2e7e844583ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-sg5b6"
Nov 03 18:00:00.327955 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.327920    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5z2z\" (UniqueName: \"kubernetes.io/projected/7f230091-46b6-4e43-9241-616506e54320-kube-api-access-g5z2z\") pod \"maas-api-key-cleanup-e2e-worker-w0-0ccbb9-29895480-wg82z\" (UID: \"7f230091-46b6-4e43-9241-616506e54320\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0ccbb9-29895480-wg82z"
Nov 03 18:00:00.328104 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.328015    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f230091-46b6-4e43-9241-616506e54320-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-0ccbb9-29895480-wg82z\" (UID: \"7f230091-46b6-4e43-9241-616506e54320\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0ccbb9-29895480-wg82z"
Nov 03 18:00:00.328346 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.328324    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8c7d4f4-916e-4418-aa0a-2e7e844583ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895480-sg5b6\" (UID: \"a8c7d4f4-916e-4418-aa0a-2e7e844583ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-sg5b6"
Nov 03 18:00:00.328437 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.328372    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/944b5b5f-42cf-4c48-8676-b36b0b4dd49f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-aab753-29895480-xznh8\" (UID: \"944b5b5f-42cf-4c48-8676-b36b0b4dd49f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895480-xznh8"
Nov 03 18:00:00.328670 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.328641    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5026ac7e-119a-4c3d-865c-848d68f35bdf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-a86228-29895480-zb2db\" (UID: \"5026ac7e-119a-4c3d-865c-848d68f35bdf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895480-zb2db"
Nov 03 18:00:00.385396 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.385326    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn5wx\" (UniqueName: \"kubernetes.io/projected/a8c7d4f4-916e-4418-aa0a-2e7e844583ba-kube-api-access-gn5wx\") pod \"maas-api-key-cleanup-29895480-sg5b6\" (UID: \"a8c7d4f4-916e-4418-aa0a-2e7e844583ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-sg5b6"
Nov 03 18:00:00.385396 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.385357    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kwwc\" (UniqueName: \"kubernetes.io/projected/944b5b5f-42cf-4c48-8676-b36b0b4dd49f-kube-api-access-8kwwc\") pod \"maas-api-key-cleanup-e2e-worker-w6-aab753-29895480-xznh8\" (UID: \"944b5b5f-42cf-4c48-8676-b36b0b4dd49f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895480-xznh8"
Nov 03 18:00:00.385575 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.385530    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkqkq\" (UniqueName: \"kubernetes.io/projected/5026ac7e-119a-4c3d-865c-848d68f35bdf-kube-api-access-hkqkq\") pod \"maas-api-key-cleanup-e2e-worker-w4-a86228-29895480-zb2db\" (UID: \"5026ac7e-119a-4c3d-865c-848d68f35bdf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895480-zb2db"
Nov 03 18:00:00.429386 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.429346    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/870683a9-9f9e-4071-a4e2-4e62a0178309-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-ef9633-29895480-v2xwh\" (UID: \"870683a9-9f9e-4071-a4e2-4e62a0178309\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895480-v2xwh"
Nov 03 18:00:00.429579 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.429398    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5z2z\" (UniqueName: \"kubernetes.io/projected/7f230091-46b6-4e43-9241-616506e54320-kube-api-access-g5z2z\") pod \"maas-api-key-cleanup-e2e-worker-w0-0ccbb9-29895480-wg82z\" (UID: \"7f230091-46b6-4e43-9241-616506e54320\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0ccbb9-29895480-wg82z"
Nov 03 18:00:00.429579 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.429435    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f230091-46b6-4e43-9241-616506e54320-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-0ccbb9-29895480-wg82z\" (UID: \"7f230091-46b6-4e43-9241-616506e54320\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0ccbb9-29895480-wg82z"
Nov 03 18:00:00.429713 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.429600    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfmsg\" (UniqueName: \"kubernetes.io/projected/870683a9-9f9e-4071-a4e2-4e62a0178309-kube-api-access-xfmsg\") pod \"maas-api-key-cleanup-e2e-worker-w1-ef9633-29895480-v2xwh\" (UID: \"870683a9-9f9e-4071-a4e2-4e62a0178309\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895480-v2xwh"
Nov 03 18:00:00.430095 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.430072    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/870683a9-9f9e-4071-a4e2-4e62a0178309-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-ef9633-29895480-v2xwh\" (UID: \"870683a9-9f9e-4071-a4e2-4e62a0178309\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895480-v2xwh"
Nov 03 18:00:00.430204 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.430186    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f230091-46b6-4e43-9241-616506e54320-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-0ccbb9-29895480-wg82z\" (UID: \"7f230091-46b6-4e43-9241-616506e54320\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0ccbb9-29895480-wg82z"
Nov 03 18:00:00.487380 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.487346    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5z2z\" (UniqueName: \"kubernetes.io/projected/7f230091-46b6-4e43-9241-616506e54320-kube-api-access-g5z2z\") pod \"maas-api-key-cleanup-e2e-worker-w0-0ccbb9-29895480-wg82z\" (UID: \"7f230091-46b6-4e43-9241-616506e54320\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0ccbb9-29895480-wg82z"
Nov 03 18:00:00.487525 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.487348    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfmsg\" (UniqueName: \"kubernetes.io/projected/870683a9-9f9e-4071-a4e2-4e62a0178309-kube-api-access-xfmsg\") pod \"maas-api-key-cleanup-e2e-worker-w1-ef9633-29895480-v2xwh\" (UID: \"870683a9-9f9e-4071-a4e2-4e62a0178309\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895480-v2xwh"
Nov 03 18:00:00.492065 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.492041    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895480-xznh8"
Nov 03 18:00:00.500315 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.500288    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895480-zb2db"
Nov 03 18:00:00.507788 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.507762    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-sg5b6"
Nov 03 18:00:00.519151 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.519123    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0ccbb9-29895480-wg82z"
Nov 03 18:00:00.528026 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.527997    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895480-v2xwh"
Nov 03 18:00:00.829546 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.829516    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d3ab78f-bvrnk"
Nov 03 18:00:00.829726 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.829538    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d3ab78f-bvrnk" event={"ID":"cf859d09-eac4-4450-9269-e8cb2c3020f5","Type":"ContainerDied","Data":"b8ede2b342bd52d2e8816d6dbeba137e6679a4d500ddc1519b304326346e1a91"}
Nov 03 18:00:00.829726 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.829594    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8ede2b342bd52d2e8816d6dbeba137e6679a4d500ddc1519b304326346e1a91"
Nov 03 18:00:00.883940 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.883911    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895480-xznh8"]
Nov 03 18:00:00.890416 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:00:00.890387    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod944b5b5f_42cf_4c48_8676_b36b0b4dd49f.slice/crio-9e91b75144ae339b599d6e07def5e6f15bc13197a2d1b32cca1c20d451670ec2 WatchSource:0}: Error finding container 9e91b75144ae339b599d6e07def5e6f15bc13197a2d1b32cca1c20d451670ec2: Status 404 returned error can't find the container with id 9e91b75144ae339b599d6e07def5e6f15bc13197a2d1b32cca1c20d451670ec2
Nov 03 18:00:00.935415 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.935386    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895480-sg5b6"]
Nov 03 18:00:00.937791 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.937758    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895480-zb2db"]
Nov 03 18:00:00.939989 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.939965    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0ccbb9-29895480-wg82z"]
Nov 03 18:00:00.944366 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:00:00.944339    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c7d4f4_916e_4418_aa0a_2e7e844583ba.slice/crio-a3a472ad4459b710af7d1cf5d44bd379dac40621fb7d9b404dd467654d4da307 WatchSource:0}: Error finding container a3a472ad4459b710af7d1cf5d44bd379dac40621fb7d9b404dd467654d4da307: Status 404 returned error can't find the container with id a3a472ad4459b710af7d1cf5d44bd379dac40621fb7d9b404dd467654d4da307
Nov 03 18:00:00.945925 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:00:00.945903    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f230091_46b6_4e43_9241_616506e54320.slice/crio-b8a02d54956aa14233ffbd8d16a0251dad2fe96a494cc8809334f6486c32e3b4 WatchSource:0}: Error finding container b8a02d54956aa14233ffbd8d16a0251dad2fe96a494cc8809334f6486c32e3b4: Status 404 returned error can't find the container with id b8a02d54956aa14233ffbd8d16a0251dad2fe96a494cc8809334f6486c32e3b4
Nov 03 18:00:00.947430 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:00:00.947404    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5026ac7e_119a_4c3d_865c_848d68f35bdf.slice/crio-78a099083aa09a4716ecc4db970119980dc4bc405fd8d39d270360a577e9510a WatchSource:0}: Error finding container 78a099083aa09a4716ecc4db970119980dc4bc405fd8d39d270360a577e9510a: Status 404 returned error can't find the container with id 78a099083aa09a4716ecc4db970119980dc4bc405fd8d39d270360a577e9510a
Nov 03 18:00:00.987400 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:00.987368    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895480-v2xwh"]
Nov 03 18:00:01.000861 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:00:01.000834    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod870683a9_9f9e_4071_a4e2_4e62a0178309.slice/crio-7f5241ac414186f0c349bd44f14b491467af9579aaaf2b44fff05bb2a9581986 WatchSource:0}: Error finding container 7f5241ac414186f0c349bd44f14b491467af9579aaaf2b44fff05bb2a9581986: Status 404 returned error can't find the container with id 7f5241ac414186f0c349bd44f14b491467af9579aaaf2b44fff05bb2a9581986
Nov 03 18:00:01.839602 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:01.839564    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0ccbb9-29895480-wg82z" event={"ID":"7f230091-46b6-4e43-9241-616506e54320","Type":"ContainerStarted","Data":"34b3da4f7f7af1b5f97459f5ebc4fed7306c43c32192ca8704431de12db76a3f"}
Nov 03 18:00:01.839778 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:01.839644    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0ccbb9-29895480-wg82z" event={"ID":"7f230091-46b6-4e43-9241-616506e54320","Type":"ContainerStarted","Data":"b8a02d54956aa14233ffbd8d16a0251dad2fe96a494cc8809334f6486c32e3b4"}
Nov 03 18:00:01.841346 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:01.841316    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895480-xznh8" event={"ID":"944b5b5f-42cf-4c48-8676-b36b0b4dd49f","Type":"ContainerStarted","Data":"75851a5dc16acca9ff6649a9b9b226ef7bcab48c92888bcbcc736699d15511b9"}
Nov 03 18:00:01.841484 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:01.841364    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895480-xznh8" event={"ID":"944b5b5f-42cf-4c48-8676-b36b0b4dd49f","Type":"ContainerStarted","Data":"9e91b75144ae339b599d6e07def5e6f15bc13197a2d1b32cca1c20d451670ec2"}
Nov 03 18:00:01.841577 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:01.841560    5315 scope.go:122] "RemoveContainer" containerID="75851a5dc16acca9ff6649a9b9b226ef7bcab48c92888bcbcc736699d15511b9"
Nov 03 18:00:01.843096 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:01.843064    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895480-v2xwh" event={"ID":"870683a9-9f9e-4071-a4e2-4e62a0178309","Type":"ContainerStarted","Data":"a23bcd9206833d7b9bf1252789520afe51d6e95194f9defaa75de7d13f21eb2c"}
Nov 03 18:00:01.843199 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:01.843128    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895480-v2xwh" event={"ID":"870683a9-9f9e-4071-a4e2-4e62a0178309","Type":"ContainerStarted","Data":"7f5241ac414186f0c349bd44f14b491467af9579aaaf2b44fff05bb2a9581986"}
Nov 03 18:00:01.845036 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:01.845002    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895480-zb2db" event={"ID":"5026ac7e-119a-4c3d-865c-848d68f35bdf","Type":"ContainerStarted","Data":"6b7a10f424296379529f36c8450a70714dd4ce6e26b0472c9f4de2a02cb7664b"}
Nov 03 18:00:01.845155 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:01.845064    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895480-zb2db" event={"ID":"5026ac7e-119a-4c3d-865c-848d68f35bdf","Type":"ContainerStarted","Data":"78a099083aa09a4716ecc4db970119980dc4bc405fd8d39d270360a577e9510a"}
Nov 03 18:00:01.846776 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:01.846746    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-sg5b6" event={"ID":"a8c7d4f4-916e-4418-aa0a-2e7e844583ba","Type":"ContainerStarted","Data":"d4e0e6b8083f56c0111824e56356903049f7341ebc31df933af2be4b18dab63a"}
Nov 03 18:00:01.846884 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:01.846800    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-sg5b6" event={"ID":"a8c7d4f4-916e-4418-aa0a-2e7e844583ba","Type":"ContainerStarted","Data":"a3a472ad4459b710af7d1cf5d44bd379dac40621fb7d9b404dd467654d4da307"}
Nov 03 18:00:01.952163 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:01.952102    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0ccbb9-29895480-wg82z" podStartSLOduration=1.95208885 podStartE2EDuration="1.95208885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:00:01.950795483 +0000 UTC m=+840.966993551" watchObservedRunningTime="2026-11-03 18:00:01.95208885 +0000 UTC m=+840.968286919"
Nov 03 18:00:02.060012 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:02.059954    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-sg5b6" podStartSLOduration=2.059939601 podStartE2EDuration="2.059939601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:00:02.058567007 +0000 UTC m=+841.074765088" watchObservedRunningTime="2026-11-03 18:00:02.059939601 +0000 UTC m=+841.076137669"
Nov 03 18:00:02.169394 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:02.169339    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895480-v2xwh" podStartSLOduration=2.169322131 podStartE2EDuration="2.169322131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:00:02.1679368 +0000 UTC m=+841.184134869" watchObservedRunningTime="2026-11-03 18:00:02.169322131 +0000 UTC m=+841.185520199"
Nov 03 18:00:02.277125 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:02.277066    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895480-zb2db" podStartSLOduration=2.277052255 podStartE2EDuration="2.277052255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:00:02.275771046 +0000 UTC m=+841.291969115" watchObservedRunningTime="2026-11-03 18:00:02.277052255 +0000 UTC m=+841.293250376"
Nov 03 18:00:02.855467 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:02.855436    5315 generic.go:356] "Generic (PLEG): container finished" podID="944b5b5f-42cf-4c48-8676-b36b0b4dd49f" containerID="75851a5dc16acca9ff6649a9b9b226ef7bcab48c92888bcbcc736699d15511b9" exitCode=60
Nov 03 18:00:02.855467 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:02.855460    5315 generic.go:356] "Generic (PLEG): container finished" podID="944b5b5f-42cf-4c48-8676-b36b0b4dd49f" containerID="0b3015e99352e6438ad71bac00f69206c5806eb5573460b9d80d7daa83f99c06" exitCode=60
Nov 03 18:00:02.855686 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:02.855517    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895480-xznh8" event={"ID":"944b5b5f-42cf-4c48-8676-b36b0b4dd49f","Type":"ContainerDied","Data":"75851a5dc16acca9ff6649a9b9b226ef7bcab48c92888bcbcc736699d15511b9"}
Nov 03 18:00:02.855686 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:02.855571    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895480-xznh8" event={"ID":"944b5b5f-42cf-4c48-8676-b36b0b4dd49f","Type":"ContainerDied","Data":"0b3015e99352e6438ad71bac00f69206c5806eb5573460b9d80d7daa83f99c06"}
Nov 03 18:00:02.855686 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:02.855667    5315 scope.go:122] "RemoveContainer" containerID="75851a5dc16acca9ff6649a9b9b226ef7bcab48c92888bcbcc736699d15511b9"
Nov 03 18:00:02.855870 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:02.855852    5315 scope.go:122] "RemoveContainer" containerID="0b3015e99352e6438ad71bac00f69206c5806eb5573460b9d80d7daa83f99c06"
Nov 03 18:00:02.856131 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:00:02.856111    5315 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-aab753-29895480-xznh8_odh-ai-gateway-infra(944b5b5f-42cf-4c48-8676-b36b0b4dd49f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895480-xznh8" podUID="944b5b5f-42cf-4c48-8676-b36b0b4dd49f"
Nov 03 18:00:02.857172 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:02.857153    5315 generic.go:356] "Generic (PLEG): container finished" podID="870683a9-9f9e-4071-a4e2-4e62a0178309" containerID="a23bcd9206833d7b9bf1252789520afe51d6e95194f9defaa75de7d13f21eb2c" exitCode=60
Nov 03 18:00:02.857274 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:02.857218    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895480-v2xwh" event={"ID":"870683a9-9f9e-4071-a4e2-4e62a0178309","Type":"ContainerDied","Data":"a23bcd9206833d7b9bf1252789520afe51d6e95194f9defaa75de7d13f21eb2c"}
Nov 03 18:00:02.857444 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:02.857431    5315 scope.go:122] "RemoveContainer" containerID="a23bcd9206833d7b9bf1252789520afe51d6e95194f9defaa75de7d13f21eb2c"
Nov 03 18:00:02.858748 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:02.858730    5315 generic.go:356] "Generic (PLEG): container finished" podID="5026ac7e-119a-4c3d-865c-848d68f35bdf" containerID="6b7a10f424296379529f36c8450a70714dd4ce6e26b0472c9f4de2a02cb7664b" exitCode=60
Nov 03 18:00:02.858847 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:02.858820    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895480-zb2db" event={"ID":"5026ac7e-119a-4c3d-865c-848d68f35bdf","Type":"ContainerDied","Data":"6b7a10f424296379529f36c8450a70714dd4ce6e26b0472c9f4de2a02cb7664b"}
Nov 03 18:00:02.858983 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:02.858965    5315 scope.go:122] "RemoveContainer" containerID="6b7a10f424296379529f36c8450a70714dd4ce6e26b0472c9f4de2a02cb7664b"
Nov 03 18:00:02.860651 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:02.860602    5315 generic.go:356] "Generic (PLEG): container finished" podID="a8c7d4f4-916e-4418-aa0a-2e7e844583ba" containerID="d4e0e6b8083f56c0111824e56356903049f7341ebc31df933af2be4b18dab63a" exitCode=60
Nov 03 18:00:02.860749 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:02.860646    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-sg5b6" event={"ID":"a8c7d4f4-916e-4418-aa0a-2e7e844583ba","Type":"ContainerDied","Data":"d4e0e6b8083f56c0111824e56356903049f7341ebc31df933af2be4b18dab63a"}
Nov 03 18:00:02.860867 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:02.860852    5315 scope.go:122] "RemoveContainer" containerID="d4e0e6b8083f56c0111824e56356903049f7341ebc31df933af2be4b18dab63a"
Nov 03 18:00:02.862380 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:02.862360    5315 generic.go:356] "Generic (PLEG): container finished" podID="7f230091-46b6-4e43-9241-616506e54320" containerID="34b3da4f7f7af1b5f97459f5ebc4fed7306c43c32192ca8704431de12db76a3f" exitCode=60
Nov 03 18:00:02.862443 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:02.862393    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0ccbb9-29895480-wg82z" event={"ID":"7f230091-46b6-4e43-9241-616506e54320","Type":"ContainerDied","Data":"34b3da4f7f7af1b5f97459f5ebc4fed7306c43c32192ca8704431de12db76a3f"}
Nov 03 18:00:02.862673 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:02.862655    5315 scope.go:122] "RemoveContainer" containerID="34b3da4f7f7af1b5f97459f5ebc4fed7306c43c32192ca8704431de12db76a3f"
Nov 03 18:00:02.902079 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:02.902053    5315 scope.go:122] "RemoveContainer" containerID="75851a5dc16acca9ff6649a9b9b226ef7bcab48c92888bcbcc736699d15511b9"
Nov 03 18:00:02.902529 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:00:02.902453    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75851a5dc16acca9ff6649a9b9b226ef7bcab48c92888bcbcc736699d15511b9\": container with ID starting with 75851a5dc16acca9ff6649a9b9b226ef7bcab48c92888bcbcc736699d15511b9 not found: ID does not exist" containerID="75851a5dc16acca9ff6649a9b9b226ef7bcab48c92888bcbcc736699d15511b9"
Nov 03 18:00:02.902529 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:02.902502    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75851a5dc16acca9ff6649a9b9b226ef7bcab48c92888bcbcc736699d15511b9"} err="failed to get container status \"75851a5dc16acca9ff6649a9b9b226ef7bcab48c92888bcbcc736699d15511b9\": rpc error: code = NotFound desc = could not find container \"75851a5dc16acca9ff6649a9b9b226ef7bcab48c92888bcbcc736699d15511b9\": container with ID starting with 75851a5dc16acca9ff6649a9b9b226ef7bcab48c92888bcbcc736699d15511b9 not found: ID does not exist"
Nov 03 18:00:03.788037 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:00:03.787995    5315 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod870683a9_9f9e_4071_a4e2_4e62a0178309.slice/crio-conmon-29c2bbf436371e5b45144cb24e6aafb19275104741e7e7e16bc9e90aa77b2fcc.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 18:00:03.869735 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:03.869703    5315 generic.go:356] "Generic (PLEG): container finished" podID="a8c7d4f4-916e-4418-aa0a-2e7e844583ba" containerID="bb2ae0aef7c903b8d9a24f42ae2d025785363276c76f4ee83a39e5cabb114b2e" exitCode=60
Nov 03 18:00:03.869886 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:03.869778    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-sg5b6" event={"ID":"a8c7d4f4-916e-4418-aa0a-2e7e844583ba","Type":"ContainerDied","Data":"bb2ae0aef7c903b8d9a24f42ae2d025785363276c76f4ee83a39e5cabb114b2e"}
Nov 03 18:00:03.869886 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:03.869838    5315 scope.go:122] "RemoveContainer" containerID="d4e0e6b8083f56c0111824e56356903049f7341ebc31df933af2be4b18dab63a"
Nov 03 18:00:03.870088 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:03.870073    5315 scope.go:122] "RemoveContainer" containerID="bb2ae0aef7c903b8d9a24f42ae2d025785363276c76f4ee83a39e5cabb114b2e"
Nov 03 18:00:03.870357 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:00:03.870336    5315 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895480-sg5b6_odh-ai-gateway-infra(a8c7d4f4-916e-4418-aa0a-2e7e844583ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-sg5b6" podUID="a8c7d4f4-916e-4418-aa0a-2e7e844583ba"
Nov 03 18:00:03.871724 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:03.871700    5315 generic.go:356] "Generic (PLEG): container finished" podID="7f230091-46b6-4e43-9241-616506e54320" containerID="aafbf8a8fd9899d6b02d01c93f07c82c5e6f0197f358102e344b648dfe3ae042" exitCode=60
Nov 03 18:00:03.871818 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:03.871727    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0ccbb9-29895480-wg82z" event={"ID":"7f230091-46b6-4e43-9241-616506e54320","Type":"ContainerDied","Data":"aafbf8a8fd9899d6b02d01c93f07c82c5e6f0197f358102e344b648dfe3ae042"}
Nov 03 18:00:03.872015 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:03.871998    5315 scope.go:122] "RemoveContainer" containerID="aafbf8a8fd9899d6b02d01c93f07c82c5e6f0197f358102e344b648dfe3ae042"
Nov 03 18:00:03.872255 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:00:03.872236    5315 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-0ccbb9-29895480-wg82z_odh-ai-gateway-infra(7f230091-46b6-4e43-9241-616506e54320)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0ccbb9-29895480-wg82z" podUID="7f230091-46b6-4e43-9241-616506e54320"
Nov 03 18:00:03.873557 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:03.873539    5315 scope.go:122] "RemoveContainer" containerID="0b3015e99352e6438ad71bac00f69206c5806eb5573460b9d80d7daa83f99c06"
Nov 03 18:00:03.873773 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:00:03.873753    5315 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-aab753-29895480-xznh8_odh-ai-gateway-infra(944b5b5f-42cf-4c48-8676-b36b0b4dd49f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895480-xznh8" podUID="944b5b5f-42cf-4c48-8676-b36b0b4dd49f"
Nov 03 18:00:03.875157 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:03.875135    5315 generic.go:356] "Generic (PLEG): container finished" podID="870683a9-9f9e-4071-a4e2-4e62a0178309" containerID="29c2bbf436371e5b45144cb24e6aafb19275104741e7e7e16bc9e90aa77b2fcc" exitCode=60
Nov 03 18:00:03.875244 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:03.875202    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895480-v2xwh" event={"ID":"870683a9-9f9e-4071-a4e2-4e62a0178309","Type":"ContainerDied","Data":"29c2bbf436371e5b45144cb24e6aafb19275104741e7e7e16bc9e90aa77b2fcc"}
Nov 03 18:00:03.875473 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:03.875453    5315 scope.go:122] "RemoveContainer" containerID="29c2bbf436371e5b45144cb24e6aafb19275104741e7e7e16bc9e90aa77b2fcc"
Nov 03 18:00:03.875752 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:00:03.875731    5315 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-ef9633-29895480-v2xwh_odh-ai-gateway-infra(870683a9-9f9e-4071-a4e2-4e62a0178309)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895480-v2xwh" podUID="870683a9-9f9e-4071-a4e2-4e62a0178309"
Nov 03 18:00:03.877078 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:03.877053    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895480-zb2db" event={"ID":"5026ac7e-119a-4c3d-865c-848d68f35bdf","Type":"ContainerStarted","Data":"73a2b3086cd9c8cdc0d9c91b96a930b2e4b72cfd882fa8419d9b91e3265e494a"}
Nov 03 18:00:04.021917 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:04.021877    5315 scope.go:122] "RemoveContainer" containerID="34b3da4f7f7af1b5f97459f5ebc4fed7306c43c32192ca8704431de12db76a3f"
Nov 03 18:00:04.036889 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:04.036865    5315 scope.go:122] "RemoveContainer" containerID="a23bcd9206833d7b9bf1252789520afe51d6e95194f9defaa75de7d13f21eb2c"
Nov 03 18:00:04.885466 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:04.885434    5315 scope.go:122] "RemoveContainer" containerID="bb2ae0aef7c903b8d9a24f42ae2d025785363276c76f4ee83a39e5cabb114b2e"
Nov 03 18:00:04.885992 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:00:04.885766    5315 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895480-sg5b6_odh-ai-gateway-infra(a8c7d4f4-916e-4418-aa0a-2e7e844583ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-sg5b6" podUID="a8c7d4f4-916e-4418-aa0a-2e7e844583ba"
Nov 03 18:00:04.887112 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:04.887091    5315 scope.go:122] "RemoveContainer" containerID="aafbf8a8fd9899d6b02d01c93f07c82c5e6f0197f358102e344b648dfe3ae042"
Nov 03 18:00:04.887357 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:00:04.887330    5315 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-0ccbb9-29895480-wg82z_odh-ai-gateway-infra(7f230091-46b6-4e43-9241-616506e54320)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0ccbb9-29895480-wg82z" podUID="7f230091-46b6-4e43-9241-616506e54320"
Nov 03 18:00:04.888800 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:04.888779    5315 scope.go:122] "RemoveContainer" containerID="29c2bbf436371e5b45144cb24e6aafb19275104741e7e7e16bc9e90aa77b2fcc"
Nov 03 18:00:04.889019 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:00:04.888998    5315 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-ef9633-29895480-v2xwh_odh-ai-gateway-infra(870683a9-9f9e-4071-a4e2-4e62a0178309)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895480-v2xwh" podUID="870683a9-9f9e-4071-a4e2-4e62a0178309"
Nov 03 18:00:04.890315 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:04.890292    5315 generic.go:356] "Generic (PLEG): container finished" podID="5026ac7e-119a-4c3d-865c-848d68f35bdf" containerID="73a2b3086cd9c8cdc0d9c91b96a930b2e4b72cfd882fa8419d9b91e3265e494a" exitCode=60
Nov 03 18:00:04.890440 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:04.890352    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895480-zb2db" event={"ID":"5026ac7e-119a-4c3d-865c-848d68f35bdf","Type":"ContainerDied","Data":"73a2b3086cd9c8cdc0d9c91b96a930b2e4b72cfd882fa8419d9b91e3265e494a"}
Nov 03 18:00:04.890440 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:04.890391    5315 scope.go:122] "RemoveContainer" containerID="6b7a10f424296379529f36c8450a70714dd4ce6e26b0472c9f4de2a02cb7664b"
Nov 03 18:00:04.890548 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:04.890521    5315 scope.go:122] "RemoveContainer" containerID="73a2b3086cd9c8cdc0d9c91b96a930b2e4b72cfd882fa8419d9b91e3265e494a"
Nov 03 18:00:04.890761 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:00:04.890741    5315 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-a86228-29895480-zb2db_odh-ai-gateway-infra(5026ac7e-119a-4c3d-865c-848d68f35bdf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895480-zb2db" podUID="5026ac7e-119a-4c3d-865c-848d68f35bdf"
Nov 03 18:00:05.897179 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:05.897144    5315 scope.go:122] "RemoveContainer" containerID="73a2b3086cd9c8cdc0d9c91b96a930b2e4b72cfd882fa8419d9b91e3265e494a"
Nov 03 18:00:05.897572 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:00:05.897379    5315 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-a86228-29895480-zb2db_odh-ai-gateway-infra(5026ac7e-119a-4c3d-865c-848d68f35bdf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895480-zb2db" podUID="5026ac7e-119a-4c3d-865c-848d68f35bdf"
Nov 03 18:00:07.217067 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:07.217029    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6af2fee0-hhwhq"]
Nov 03 18:00:07.222360 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:07.222337    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af2fee0-hhwhq"
Nov 03 18:00:07.271754 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:07.271720    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:00:07.301940 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:07.301909    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/590eae70-e712-474f-8b0c-dc6cb1873e63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6af2fee0-hhwhq\" (UID: \"590eae70-e712-474f-8b0c-dc6cb1873e63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af2fee0-hhwhq"
Nov 03 18:00:07.302136 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:07.301970    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/590eae70-e712-474f-8b0c-dc6cb1873e63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6af2fee0-hhwhq\" (UID: \"590eae70-e712-474f-8b0c-dc6cb1873e63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af2fee0-hhwhq"
Nov 03 18:00:07.302206 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:07.302142    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6clhk\" (UniqueName: \"kubernetes.io/projected/590eae70-e712-474f-8b0c-dc6cb1873e63-kube-api-access-6clhk\") pod \"maas-api-revoke-sub-6af2fee0-hhwhq\" (UID: \"590eae70-e712-474f-8b0c-dc6cb1873e63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af2fee0-hhwhq"
Nov 03 18:00:07.319705 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:07.319672    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6af2fee0-hhwhq"]
Nov 03 18:00:07.403091 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:07.403052    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/590eae70-e712-474f-8b0c-dc6cb1873e63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6af2fee0-hhwhq\" (UID: \"590eae70-e712-474f-8b0c-dc6cb1873e63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af2fee0-hhwhq"
Nov 03 18:00:07.403269 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:07.403114    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/590eae70-e712-474f-8b0c-dc6cb1873e63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6af2fee0-hhwhq\" (UID: \"590eae70-e712-474f-8b0c-dc6cb1873e63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af2fee0-hhwhq"
Nov 03 18:00:07.403269 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:07.403180    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6clhk\" (UniqueName: \"kubernetes.io/projected/590eae70-e712-474f-8b0c-dc6cb1873e63-kube-api-access-6clhk\") pod \"maas-api-revoke-sub-6af2fee0-hhwhq\" (UID: \"590eae70-e712-474f-8b0c-dc6cb1873e63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af2fee0-hhwhq"
Nov 03 18:00:07.403803 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:07.403774    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/590eae70-e712-474f-8b0c-dc6cb1873e63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6af2fee0-hhwhq\" (UID: \"590eae70-e712-474f-8b0c-dc6cb1873e63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af2fee0-hhwhq"
Nov 03 18:00:07.405419 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:07.405394    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/590eae70-e712-474f-8b0c-dc6cb1873e63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6af2fee0-hhwhq\" (UID: \"590eae70-e712-474f-8b0c-dc6cb1873e63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af2fee0-hhwhq"
Nov 03 18:00:07.460970 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:07.460935    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6clhk\" (UniqueName: \"kubernetes.io/projected/590eae70-e712-474f-8b0c-dc6cb1873e63-kube-api-access-6clhk\") pod \"maas-api-revoke-sub-6af2fee0-hhwhq\" (UID: \"590eae70-e712-474f-8b0c-dc6cb1873e63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af2fee0-hhwhq"
Nov 03 18:00:07.535370 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:07.535337    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af2fee0-hhwhq"
Nov 03 18:00:07.941682 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:07.941578    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6af2fee0-hhwhq"]
Nov 03 18:00:07.949111 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:00:07.949074    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod590eae70_e712_474f_8b0c_dc6cb1873e63.slice/crio-8639c442bd4954b351aea1a201ba462e29bb063bb378e7413b67a54eadb6f90e WatchSource:0}: Error finding container 8639c442bd4954b351aea1a201ba462e29bb063bb378e7413b67a54eadb6f90e: Status 404 returned error can't find the container with id 8639c442bd4954b351aea1a201ba462e29bb063bb378e7413b67a54eadb6f90e
Nov 03 18:00:08.914590 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:08.914552    5315 generic.go:356] "Generic (PLEG): container finished" podID="590eae70-e712-474f-8b0c-dc6cb1873e63" containerID="d450a6f5c7cdec09d0f499c4e4b1c34562b6d471544082e953012d3af44ad3e8" exitCode=0
Nov 03 18:00:08.914987 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:08.914646    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af2fee0-hhwhq" event={"ID":"590eae70-e712-474f-8b0c-dc6cb1873e63","Type":"ContainerDied","Data":"d450a6f5c7cdec09d0f499c4e4b1c34562b6d471544082e953012d3af44ad3e8"}
Nov 03 18:00:08.914987 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:08.914731    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af2fee0-hhwhq" event={"ID":"590eae70-e712-474f-8b0c-dc6cb1873e63","Type":"ContainerStarted","Data":"8639c442bd4954b351aea1a201ba462e29bb063bb378e7413b67a54eadb6f90e"}
Nov 03 18:00:10.070261 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:10.070234    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af2fee0-hhwhq"
Nov 03 18:00:10.232088 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:10.232003    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/590eae70-e712-474f-8b0c-dc6cb1873e63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/590eae70-e712-474f-8b0c-dc6cb1873e63-maas-api-xks-ca\") pod \"590eae70-e712-474f-8b0c-dc6cb1873e63\" (UID: \"590eae70-e712-474f-8b0c-dc6cb1873e63\") "
Nov 03 18:00:10.232088 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:10.232055    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/590eae70-e712-474f-8b0c-dc6cb1873e63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/590eae70-e712-474f-8b0c-dc6cb1873e63-maas-api-openshift-ca\") pod \"590eae70-e712-474f-8b0c-dc6cb1873e63\" (UID: \"590eae70-e712-474f-8b0c-dc6cb1873e63\") "
Nov 03 18:00:10.232326 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:10.232171    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/590eae70-e712-474f-8b0c-dc6cb1873e63-kube-api-access-6clhk\" (UniqueName: \"kubernetes.io/projected/590eae70-e712-474f-8b0c-dc6cb1873e63-kube-api-access-6clhk\") pod \"590eae70-e712-474f-8b0c-dc6cb1873e63\" (UID: \"590eae70-e712-474f-8b0c-dc6cb1873e63\") "
Nov 03 18:00:10.232447 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:10.232423    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/590eae70-e712-474f-8b0c-dc6cb1873e63-maas-api-openshift-ca" pod "590eae70-e712-474f-8b0c-dc6cb1873e63" (UID: "590eae70-e712-474f-8b0c-dc6cb1873e63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:00:10.234203 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:10.234180    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/590eae70-e712-474f-8b0c-dc6cb1873e63-maas-api-xks-ca" pod "590eae70-e712-474f-8b0c-dc6cb1873e63" (UID: "590eae70-e712-474f-8b0c-dc6cb1873e63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:00:10.234270 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:10.234235    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/590eae70-e712-474f-8b0c-dc6cb1873e63-kube-api-access-6clhk" pod "590eae70-e712-474f-8b0c-dc6cb1873e63" (UID: "590eae70-e712-474f-8b0c-dc6cb1873e63"). InnerVolumeSpecName "kube-api-access-6clhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:00:10.333598 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:10.333564    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/590eae70-e712-474f-8b0c-dc6cb1873e63-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:00:10.333598 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:10.333596    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/590eae70-e712-474f-8b0c-dc6cb1873e63-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:00:10.333832 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:10.333630    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6clhk\" (UniqueName: \"kubernetes.io/projected/590eae70-e712-474f-8b0c-dc6cb1873e63-kube-api-access-6clhk\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:00:10.928114 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:10.928081    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af2fee0-hhwhq"
Nov 03 18:00:10.928262 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:10.928109    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af2fee0-hhwhq" event={"ID":"590eae70-e712-474f-8b0c-dc6cb1873e63","Type":"ContainerDied","Data":"8639c442bd4954b351aea1a201ba462e29bb063bb378e7413b67a54eadb6f90e"}
Nov 03 18:00:10.928262 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:10.928159    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8639c442bd4954b351aea1a201ba462e29bb063bb378e7413b67a54eadb6f90e"
Nov 03 18:00:14.097264 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:14.097224    5315 scope.go:122] "RemoveContainer" containerID="0b3015e99352e6438ad71bac00f69206c5806eb5573460b9d80d7daa83f99c06"
Nov 03 18:00:15.035284 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:15.035246    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895480-xznh8" event={"ID":"944b5b5f-42cf-4c48-8676-b36b0b4dd49f","Type":"ContainerStarted","Data":"8b6edf08b01349928ca43f92b65e5d068861c3a5c03df735b32a14cec7192eeb"}
Nov 03 18:00:15.153633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:15.148993    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895480-xznh8" podStartSLOduration=15.148974582 podStartE2EDuration="15.148974582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:00:15.148156728 +0000 UTC m=+854.164354795" watchObservedRunningTime="2026-11-03 18:00:15.148974582 +0000 UTC m=+854.165172652"
Nov 03 18:00:16.042316 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:16.042281    5315 generic.go:356] "Generic (PLEG): container finished" podID="944b5b5f-42cf-4c48-8676-b36b0b4dd49f" containerID="8b6edf08b01349928ca43f92b65e5d068861c3a5c03df735b32a14cec7192eeb" exitCode=60
Nov 03 18:00:16.042495 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:16.042359    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895480-xznh8" event={"ID":"944b5b5f-42cf-4c48-8676-b36b0b4dd49f","Type":"ContainerDied","Data":"8b6edf08b01349928ca43f92b65e5d068861c3a5c03df735b32a14cec7192eeb"}
Nov 03 18:00:16.042495 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:16.042431    5315 scope.go:122] "RemoveContainer" containerID="0b3015e99352e6438ad71bac00f69206c5806eb5573460b9d80d7daa83f99c06"
Nov 03 18:00:16.042838 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:16.042820    5315 scope.go:122] "RemoveContainer" containerID="8b6edf08b01349928ca43f92b65e5d068861c3a5c03df735b32a14cec7192eeb"
Nov 03 18:00:16.043118 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:00:16.043093    5315 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-aab753-29895480-xznh8_odh-ai-gateway-infra(944b5b5f-42cf-4c48-8676-b36b0b4dd49f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895480-xznh8" podUID="944b5b5f-42cf-4c48-8676-b36b0b4dd49f"
Nov 03 18:00:16.167713 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:16.167674    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895480-xznh8"]
Nov 03 18:00:17.097259 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:17.097229    5315 scope.go:122] "RemoveContainer" containerID="73a2b3086cd9c8cdc0d9c91b96a930b2e4b72cfd882fa8419d9b91e3265e494a"
Nov 03 18:00:17.227038 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:17.227007    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895480-xznh8"
Nov 03 18:00:17.325675 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:17.325582    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/944b5b5f-42cf-4c48-8676-b36b0b4dd49f-kube-api-access-8kwwc\" (UniqueName: \"kubernetes.io/projected/944b5b5f-42cf-4c48-8676-b36b0b4dd49f-kube-api-access-8kwwc\") pod \"944b5b5f-42cf-4c48-8676-b36b0b4dd49f\" (UID: \"944b5b5f-42cf-4c48-8676-b36b0b4dd49f\") "
Nov 03 18:00:17.325877 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:17.325701    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/944b5b5f-42cf-4c48-8676-b36b0b4dd49f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/944b5b5f-42cf-4c48-8676-b36b0b4dd49f-maas-api-service-ca\") pod \"944b5b5f-42cf-4c48-8676-b36b0b4dd49f\" (UID: \"944b5b5f-42cf-4c48-8676-b36b0b4dd49f\") "
Nov 03 18:00:17.326443 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:17.326414    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/944b5b5f-42cf-4c48-8676-b36b0b4dd49f-maas-api-service-ca" pod "944b5b5f-42cf-4c48-8676-b36b0b4dd49f" (UID: "944b5b5f-42cf-4c48-8676-b36b0b4dd49f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:00:17.329364 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:17.329334    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/944b5b5f-42cf-4c48-8676-b36b0b4dd49f-kube-api-access-8kwwc" pod "944b5b5f-42cf-4c48-8676-b36b0b4dd49f" (UID: "944b5b5f-42cf-4c48-8676-b36b0b4dd49f"). InnerVolumeSpecName "kube-api-access-8kwwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:00:17.426795 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:17.426757    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kwwc\" (UniqueName: \"kubernetes.io/projected/944b5b5f-42cf-4c48-8676-b36b0b4dd49f-kube-api-access-8kwwc\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:00:17.426795 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:17.426791    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/944b5b5f-42cf-4c48-8676-b36b0b4dd49f-maas-api-service-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:00:17.549397 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:17.549366    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50b3a56f-zf47h"]
Nov 03 18:00:17.549905 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:17.549886    5315 cpu_manager.go:414] "containerMap: removing container" podUID="944b5b5f-42cf-4c48-8676-b36b0b4dd49f" containerName="cleanup"
Nov 03 18:00:17.550016 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:17.549906    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="944b5b5f-42cf-4c48-8676-b36b0b4dd49f" containerName="cleanup"
Nov 03 18:00:17.550016 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:17.549928    5315 cpu_manager.go:414] "containerMap: removing container" podUID="590eae70-e712-474f-8b0c-dc6cb1873e63" containerName="revoke-keys"
Nov 03 18:00:17.550016 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:17.549935    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="590eae70-e712-474f-8b0c-dc6cb1873e63" containerName="revoke-keys"
Nov 03 18:00:17.550016 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:17.549986    5315 cpu_manager.go:414] "containerMap: removing container" podUID="944b5b5f-42cf-4c48-8676-b36b0b4dd49f" containerName="cleanup"
Nov 03 18:00:17.550016 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:17.549994    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="944b5b5f-42cf-4c48-8676-b36b0b4dd49f" containerName="cleanup"
Nov 03 18:00:17.550254 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:17.550110    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="590eae70-e712-474f-8b0c-dc6cb1873e63" containerName="revoke-keys"
Nov 03 18:00:17.550254 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:17.550128    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="944b5b5f-42cf-4c48-8676-b36b0b4dd49f" containerName="cleanup"
Nov 03 18:00:17.550254 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:17.550138    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="944b5b5f-42cf-4c48-8676-b36b0b4dd49f" containerName="cleanup"
Nov 03 18:00:17.550254 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:17.550150    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="944b5b5f-42cf-4c48-8676-b36b0b4dd49f" containerName="cleanup"
Nov 03 18:00:17.553434 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:17.553413    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b3a56f-zf47h"
Nov 03 18:00:17.605990 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:17.605904    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:00:17.614850 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:17.614823    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50b3a56f-zf47h"]
Nov 03 18:00:17.730064 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:17.730023    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c493b7e-6578-4e59-bcd5-46e7e477a1f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50b3a56f-zf47h\" (UID: \"4c493b7e-6578-4e59-bcd5-46e7e477a1f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b3a56f-zf47h"
Nov 03 18:00:17.730234 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:17.730181    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c493b7e-6578-4e59-bcd5-46e7e477a1f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50b3a56f-zf47h\" (UID: \"4c493b7e-6578-4e59-bcd5-46e7e477a1f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b3a56f-zf47h"
Nov 03 18:00:17.730315 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:17.730286    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdqbq\" (UniqueName: \"kubernetes.io/projected/4c493b7e-6578-4e59-bcd5-46e7e477a1f0-kube-api-access-hdqbq\") pod \"maas-api-revoke-sub-50b3a56f-zf47h\" (UID: \"4c493b7e-6578-4e59-bcd5-46e7e477a1f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b3a56f-zf47h"
Nov 03 18:00:17.830897 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:17.830865    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdqbq\" (UniqueName: \"kubernetes.io/projected/4c493b7e-6578-4e59-bcd5-46e7e477a1f0-kube-api-access-hdqbq\") pod \"maas-api-revoke-sub-50b3a56f-zf47h\" (UID: \"4c493b7e-6578-4e59-bcd5-46e7e477a1f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b3a56f-zf47h"
Nov 03 18:00:17.831051 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:17.830937    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c493b7e-6578-4e59-bcd5-46e7e477a1f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50b3a56f-zf47h\" (UID: \"4c493b7e-6578-4e59-bcd5-46e7e477a1f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b3a56f-zf47h"
Nov 03 18:00:17.831051 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:17.831026    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c493b7e-6578-4e59-bcd5-46e7e477a1f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50b3a56f-zf47h\" (UID: \"4c493b7e-6578-4e59-bcd5-46e7e477a1f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b3a56f-zf47h"
Nov 03 18:00:17.831587 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:17.831561    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c493b7e-6578-4e59-bcd5-46e7e477a1f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50b3a56f-zf47h\" (UID: \"4c493b7e-6578-4e59-bcd5-46e7e477a1f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b3a56f-zf47h"
Nov 03 18:00:17.833344 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:17.833309    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c493b7e-6578-4e59-bcd5-46e7e477a1f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50b3a56f-zf47h\" (UID: \"4c493b7e-6578-4e59-bcd5-46e7e477a1f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b3a56f-zf47h"
Nov 03 18:00:17.893823 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:17.893744    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdqbq\" (UniqueName: \"kubernetes.io/projected/4c493b7e-6578-4e59-bcd5-46e7e477a1f0-kube-api-access-hdqbq\") pod \"maas-api-revoke-sub-50b3a56f-zf47h\" (UID: \"4c493b7e-6578-4e59-bcd5-46e7e477a1f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b3a56f-zf47h"
Nov 03 18:00:18.056804 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:18.056761    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895480-zb2db" event={"ID":"5026ac7e-119a-4c3d-865c-848d68f35bdf","Type":"ContainerStarted","Data":"afac5a1d181012e4bb46cf0c9b0a92fb7b4d5d263d924ec4b1ef4fd7db6b8c5a"}
Nov 03 18:00:18.058315 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:18.058287    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895480-xznh8" event={"ID":"944b5b5f-42cf-4c48-8676-b36b0b4dd49f","Type":"ContainerDied","Data":"9e91b75144ae339b599d6e07def5e6f15bc13197a2d1b32cca1c20d451670ec2"}
Nov 03 18:00:18.058448 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:18.058328    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895480-xznh8"
Nov 03 18:00:18.058448 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:18.058348    5315 scope.go:122] "RemoveContainer" containerID="8b6edf08b01349928ca43f92b65e5d068861c3a5c03df735b32a14cec7192eeb"
Nov 03 18:00:18.096982 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:18.096942    5315 scope.go:122] "RemoveContainer" containerID="bb2ae0aef7c903b8d9a24f42ae2d025785363276c76f4ee83a39e5cabb114b2e"
Nov 03 18:00:18.097125 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:18.097087    5315 scope.go:122] "RemoveContainer" containerID="aafbf8a8fd9899d6b02d01c93f07c82c5e6f0197f358102e344b648dfe3ae042"
Nov 03 18:00:18.097307 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:18.097292    5315 scope.go:122] "RemoveContainer" containerID="29c2bbf436371e5b45144cb24e6aafb19275104741e7e7e16bc9e90aa77b2fcc"
Nov 03 18:00:18.169701 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:18.169556    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b3a56f-zf47h"
Nov 03 18:00:18.274687 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:18.274638    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895480-xznh8"]
Nov 03 18:00:18.285224 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:18.285186    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895480-xznh8"]
Nov 03 18:00:18.546965 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:18.546937    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50b3a56f-zf47h"]
Nov 03 18:00:18.626162 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:00:18.626131    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c493b7e_6578_4e59_bcd5_46e7e477a1f0.slice/crio-ec12fb1773cf8b029d341fe2a8a643286815a0c600ece0a95699cabc580f80a3 WatchSource:0}: Error finding container ec12fb1773cf8b029d341fe2a8a643286815a0c600ece0a95699cabc580f80a3: Status 404 returned error can't find the container with id ec12fb1773cf8b029d341fe2a8a643286815a0c600ece0a95699cabc580f80a3
Nov 03 18:00:19.065898 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:19.065855    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0ccbb9-29895480-wg82z" event={"ID":"7f230091-46b6-4e43-9241-616506e54320","Type":"ContainerStarted","Data":"cad629f85d0e479807faee6e0213a3d0355814dd7b148caeb060974022561685"}
Nov 03 18:00:19.068583 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:19.068552    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895480-v2xwh" event={"ID":"870683a9-9f9e-4071-a4e2-4e62a0178309","Type":"ContainerStarted","Data":"4969327b29d55c34b91c86665f96f6ede2198905f5d48229b3246ff97226f336"}
Nov 03 18:00:19.070382 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:19.070356    5315 generic.go:356] "Generic (PLEG): container finished" podID="5026ac7e-119a-4c3d-865c-848d68f35bdf" containerID="afac5a1d181012e4bb46cf0c9b0a92fb7b4d5d263d924ec4b1ef4fd7db6b8c5a" exitCode=60
Nov 03 18:00:19.070481 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:19.070421    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895480-zb2db" event={"ID":"5026ac7e-119a-4c3d-865c-848d68f35bdf","Type":"ContainerDied","Data":"afac5a1d181012e4bb46cf0c9b0a92fb7b4d5d263d924ec4b1ef4fd7db6b8c5a"}
Nov 03 18:00:19.070481 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:19.070472    5315 scope.go:122] "RemoveContainer" containerID="73a2b3086cd9c8cdc0d9c91b96a930b2e4b72cfd882fa8419d9b91e3265e494a"
Nov 03 18:00:19.070809 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:19.070793    5315 scope.go:122] "RemoveContainer" containerID="afac5a1d181012e4bb46cf0c9b0a92fb7b4d5d263d924ec4b1ef4fd7db6b8c5a"
Nov 03 18:00:19.071053 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:00:19.071028    5315 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-a86228-29895480-zb2db_odh-ai-gateway-infra(5026ac7e-119a-4c3d-865c-848d68f35bdf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895480-zb2db" podUID="5026ac7e-119a-4c3d-865c-848d68f35bdf"
Nov 03 18:00:19.072566 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:19.072543    5315 generic.go:356] "Generic (PLEG): container finished" podID="a8c7d4f4-916e-4418-aa0a-2e7e844583ba" containerID="dd00494f4a89bd65a93ada6f5f055c58fada90293f3ec2266df6e83ee1451a7c" exitCode=60
Nov 03 18:00:19.072704 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:19.072651    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-sg5b6" event={"ID":"a8c7d4f4-916e-4418-aa0a-2e7e844583ba","Type":"ContainerDied","Data":"dd00494f4a89bd65a93ada6f5f055c58fada90293f3ec2266df6e83ee1451a7c"}
Nov 03 18:00:19.072981 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:19.072965    5315 scope.go:122] "RemoveContainer" containerID="dd00494f4a89bd65a93ada6f5f055c58fada90293f3ec2266df6e83ee1451a7c"
Nov 03 18:00:19.073173 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:00:19.073153    5315 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895480-sg5b6_odh-ai-gateway-infra(a8c7d4f4-916e-4418-aa0a-2e7e844583ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-sg5b6" podUID="a8c7d4f4-916e-4418-aa0a-2e7e844583ba"
Nov 03 18:00:19.074396 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:19.074362    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b3a56f-zf47h" event={"ID":"4c493b7e-6578-4e59-bcd5-46e7e477a1f0","Type":"ContainerStarted","Data":"160693fb5c554625f74c24e63a849d26f7816d3d6f4ab3f3fb1be38025ef210b"}
Nov 03 18:00:19.074493 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:19.074406    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b3a56f-zf47h" event={"ID":"4c493b7e-6578-4e59-bcd5-46e7e477a1f0","Type":"ContainerStarted","Data":"ec12fb1773cf8b029d341fe2a8a643286815a0c600ece0a95699cabc580f80a3"}
Nov 03 18:00:19.090767 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:19.090740    5315 scope.go:122] "RemoveContainer" containerID="bb2ae0aef7c903b8d9a24f42ae2d025785363276c76f4ee83a39e5cabb114b2e"
Nov 03 18:00:19.107694 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:19.106784    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="944b5b5f-42cf-4c48-8676-b36b0b4dd49f" path="/var/lib/kubelet/pods/944b5b5f-42cf-4c48-8676-b36b0b4dd49f/volumes"
Nov 03 18:00:19.183818 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:19.183784    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895480-zb2db"]
Nov 03 18:00:19.516517 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:19.516416    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b3a56f-zf47h" podStartSLOduration=2.51640137 podStartE2EDuration="2.51640137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:00:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:00:19.515106398 +0000 UTC m=+858.531304467" watchObservedRunningTime="2026-11-03 18:00:19.51640137 +0000 UTC m=+858.532599439"
Nov 03 18:00:20.083227 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:20.083194    5315 generic.go:356] "Generic (PLEG): container finished" podID="870683a9-9f9e-4071-a4e2-4e62a0178309" containerID="4969327b29d55c34b91c86665f96f6ede2198905f5d48229b3246ff97226f336" exitCode=60
Nov 03 18:00:20.083398 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:20.083268    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895480-v2xwh" event={"ID":"870683a9-9f9e-4071-a4e2-4e62a0178309","Type":"ContainerDied","Data":"4969327b29d55c34b91c86665f96f6ede2198905f5d48229b3246ff97226f336"}
Nov 03 18:00:20.083398 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:20.083337    5315 scope.go:122] "RemoveContainer" containerID="29c2bbf436371e5b45144cb24e6aafb19275104741e7e7e16bc9e90aa77b2fcc"
Nov 03 18:00:20.083803 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:20.083778    5315 scope.go:122] "RemoveContainer" containerID="4969327b29d55c34b91c86665f96f6ede2198905f5d48229b3246ff97226f336"
Nov 03 18:00:20.084130 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:00:20.084094    5315 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-ef9633-29895480-v2xwh_odh-ai-gateway-infra(870683a9-9f9e-4071-a4e2-4e62a0178309)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895480-v2xwh" podUID="870683a9-9f9e-4071-a4e2-4e62a0178309"
Nov 03 18:00:20.089250 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:20.089228    5315 generic.go:356] "Generic (PLEG): container finished" podID="4c493b7e-6578-4e59-bcd5-46e7e477a1f0" containerID="160693fb5c554625f74c24e63a849d26f7816d3d6f4ab3f3fb1be38025ef210b" exitCode=0
Nov 03 18:00:20.089374 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:20.089314    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b3a56f-zf47h" event={"ID":"4c493b7e-6578-4e59-bcd5-46e7e477a1f0","Type":"ContainerDied","Data":"160693fb5c554625f74c24e63a849d26f7816d3d6f4ab3f3fb1be38025ef210b"}
Nov 03 18:00:20.091569 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:20.091545    5315 generic.go:356] "Generic (PLEG): container finished" podID="7f230091-46b6-4e43-9241-616506e54320" containerID="cad629f85d0e479807faee6e0213a3d0355814dd7b148caeb060974022561685" exitCode=60
Nov 03 18:00:20.091682 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:20.091601    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0ccbb9-29895480-wg82z" event={"ID":"7f230091-46b6-4e43-9241-616506e54320","Type":"ContainerDied","Data":"cad629f85d0e479807faee6e0213a3d0355814dd7b148caeb060974022561685"}
Nov 03 18:00:20.092223 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:20.092203    5315 scope.go:122] "RemoveContainer" containerID="cad629f85d0e479807faee6e0213a3d0355814dd7b148caeb060974022561685"
Nov 03 18:00:20.092559 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:00:20.092532    5315 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-0ccbb9-29895480-wg82z_odh-ai-gateway-infra(7f230091-46b6-4e43-9241-616506e54320)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0ccbb9-29895480-wg82z" podUID="7f230091-46b6-4e43-9241-616506e54320"
Nov 03 18:00:20.097605 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:20.097584    5315 scope.go:122] "RemoveContainer" containerID="aafbf8a8fd9899d6b02d01c93f07c82c5e6f0197f358102e344b648dfe3ae042"
Nov 03 18:00:20.191934 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:20.191898    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0ccbb9-29895480-wg82z"]
Nov 03 18:00:20.232694 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:20.232661    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895480-zb2db"
Nov 03 18:00:20.298159 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:20.298118    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895480-sg5b6"]
Nov 03 18:00:20.362454 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:20.362423    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5026ac7e-119a-4c3d-865c-848d68f35bdf-kube-api-access-hkqkq\" (UniqueName: \"kubernetes.io/projected/5026ac7e-119a-4c3d-865c-848d68f35bdf-kube-api-access-hkqkq\") pod \"5026ac7e-119a-4c3d-865c-848d68f35bdf\" (UID: \"5026ac7e-119a-4c3d-865c-848d68f35bdf\") "
Nov 03 18:00:20.362647 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:20.362538    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5026ac7e-119a-4c3d-865c-848d68f35bdf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5026ac7e-119a-4c3d-865c-848d68f35bdf-maas-api-service-ca\") pod \"5026ac7e-119a-4c3d-865c-848d68f35bdf\" (UID: \"5026ac7e-119a-4c3d-865c-848d68f35bdf\") "
Nov 03 18:00:20.364089 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:20.363352    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5026ac7e-119a-4c3d-865c-848d68f35bdf-maas-api-service-ca" pod "5026ac7e-119a-4c3d-865c-848d68f35bdf" (UID: "5026ac7e-119a-4c3d-865c-848d68f35bdf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:00:20.366021 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:20.365985    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5026ac7e-119a-4c3d-865c-848d68f35bdf-kube-api-access-hkqkq" pod "5026ac7e-119a-4c3d-865c-848d68f35bdf" (UID: "5026ac7e-119a-4c3d-865c-848d68f35bdf"). InnerVolumeSpecName "kube-api-access-hkqkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:00:20.406663 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:20.406630    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895480-v2xwh"]
Nov 03 18:00:20.455696 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:20.455670    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-sg5b6"
Nov 03 18:00:20.464453 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:20.464427    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5026ac7e-119a-4c3d-865c-848d68f35bdf-maas-api-service-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:00:20.464453 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:20.464452    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkqkq\" (UniqueName: \"kubernetes.io/projected/5026ac7e-119a-4c3d-865c-848d68f35bdf-kube-api-access-hkqkq\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:00:20.565718 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:20.565684    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8c7d4f4-916e-4418-aa0a-2e7e844583ba-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8c7d4f4-916e-4418-aa0a-2e7e844583ba-maas-api-service-ca\") pod \"a8c7d4f4-916e-4418-aa0a-2e7e844583ba\" (UID: \"a8c7d4f4-916e-4418-aa0a-2e7e844583ba\") "
Nov 03 18:00:20.566100 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:20.565860    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8c7d4f4-916e-4418-aa0a-2e7e844583ba-kube-api-access-gn5wx\" (UniqueName: \"kubernetes.io/projected/a8c7d4f4-916e-4418-aa0a-2e7e844583ba-kube-api-access-gn5wx\") pod \"a8c7d4f4-916e-4418-aa0a-2e7e844583ba\" (UID: \"a8c7d4f4-916e-4418-aa0a-2e7e844583ba\") "
Nov 03 18:00:20.566138 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:20.566091    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8c7d4f4-916e-4418-aa0a-2e7e844583ba-maas-api-service-ca" pod "a8c7d4f4-916e-4418-aa0a-2e7e844583ba" (UID: "a8c7d4f4-916e-4418-aa0a-2e7e844583ba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:00:20.566250 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:20.566230    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8c7d4f4-916e-4418-aa0a-2e7e844583ba-maas-api-service-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:00:20.567875 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:20.567852    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8c7d4f4-916e-4418-aa0a-2e7e844583ba-kube-api-access-gn5wx" pod "a8c7d4f4-916e-4418-aa0a-2e7e844583ba" (UID: "a8c7d4f4-916e-4418-aa0a-2e7e844583ba"). InnerVolumeSpecName "kube-api-access-gn5wx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:00:20.667869 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:20.667782    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn5wx\" (UniqueName: \"kubernetes.io/projected/a8c7d4f4-916e-4418-aa0a-2e7e844583ba-kube-api-access-gn5wx\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:00:21.102225 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:21.101657    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895480-zb2db"
Nov 03 18:00:21.103405 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:21.103378    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-sg5b6"
Nov 03 18:00:21.105868 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:21.105821    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895480-zb2db" event={"ID":"5026ac7e-119a-4c3d-865c-848d68f35bdf","Type":"ContainerDied","Data":"78a099083aa09a4716ecc4db970119980dc4bc405fd8d39d270360a577e9510a"}
Nov 03 18:00:21.105982 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:21.105888    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-sg5b6" event={"ID":"a8c7d4f4-916e-4418-aa0a-2e7e844583ba","Type":"ContainerDied","Data":"a3a472ad4459b710af7d1cf5d44bd379dac40621fb7d9b404dd467654d4da307"}
Nov 03 18:00:21.105982 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:21.105922    5315 scope.go:122] "RemoveContainer" containerID="afac5a1d181012e4bb46cf0c9b0a92fb7b4d5d263d924ec4b1ef4fd7db6b8c5a"
Nov 03 18:00:21.125536 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:21.125514    5315 scope.go:122] "RemoveContainer" containerID="dd00494f4a89bd65a93ada6f5f055c58fada90293f3ec2266df6e83ee1451a7c"
Nov 03 18:00:21.255393 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:21.255358    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0ccbb9-29895480-wg82z"
Nov 03 18:00:21.256337 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:21.256313    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895480-sg5b6"]
Nov 03 18:00:21.259860 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:21.259835    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895480-sg5b6"]
Nov 03 18:00:21.293977 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:21.293949    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b3a56f-zf47h"
Nov 03 18:00:21.309894 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:21.309862    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895480-v2xwh"
Nov 03 18:00:21.377571 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:21.377469    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c493b7e-6578-4e59-bcd5-46e7e477a1f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c493b7e-6578-4e59-bcd5-46e7e477a1f0-maas-api-openshift-ca\") pod \"4c493b7e-6578-4e59-bcd5-46e7e477a1f0\" (UID: \"4c493b7e-6578-4e59-bcd5-46e7e477a1f0\") "
Nov 03 18:00:21.377763 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:21.377566    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c493b7e-6578-4e59-bcd5-46e7e477a1f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c493b7e-6578-4e59-bcd5-46e7e477a1f0-maas-api-xks-ca\") pod \"4c493b7e-6578-4e59-bcd5-46e7e477a1f0\" (UID: \"4c493b7e-6578-4e59-bcd5-46e7e477a1f0\") "
Nov 03 18:00:21.377763 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:21.377627    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f230091-46b6-4e43-9241-616506e54320-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f230091-46b6-4e43-9241-616506e54320-maas-api-service-ca\") pod \"7f230091-46b6-4e43-9241-616506e54320\" (UID: \"7f230091-46b6-4e43-9241-616506e54320\") "
Nov 03 18:00:21.377763 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:21.377668    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c493b7e-6578-4e59-bcd5-46e7e477a1f0-kube-api-access-hdqbq\" (UniqueName: \"kubernetes.io/projected/4c493b7e-6578-4e59-bcd5-46e7e477a1f0-kube-api-access-hdqbq\") pod \"4c493b7e-6578-4e59-bcd5-46e7e477a1f0\" (UID: \"4c493b7e-6578-4e59-bcd5-46e7e477a1f0\") "
Nov 03 18:00:21.377987 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:21.377804    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f230091-46b6-4e43-9241-616506e54320-kube-api-access-g5z2z\" (UniqueName: \"kubernetes.io/projected/7f230091-46b6-4e43-9241-616506e54320-kube-api-access-g5z2z\") pod \"7f230091-46b6-4e43-9241-616506e54320\" (UID: \"7f230091-46b6-4e43-9241-616506e54320\") "
Nov 03 18:00:21.377987 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:21.377891    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/870683a9-9f9e-4071-a4e2-4e62a0178309-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/870683a9-9f9e-4071-a4e2-4e62a0178309-maas-api-service-ca\") pod \"870683a9-9f9e-4071-a4e2-4e62a0178309\" (UID: \"870683a9-9f9e-4071-a4e2-4e62a0178309\") "
Nov 03 18:00:21.377987 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:21.377927    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/870683a9-9f9e-4071-a4e2-4e62a0178309-kube-api-access-xfmsg\" (UniqueName: \"kubernetes.io/projected/870683a9-9f9e-4071-a4e2-4e62a0178309-kube-api-access-xfmsg\") pod \"870683a9-9f9e-4071-a4e2-4e62a0178309\" (UID: \"870683a9-9f9e-4071-a4e2-4e62a0178309\") "
Nov 03 18:00:21.378907 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:21.378879    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c493b7e-6578-4e59-bcd5-46e7e477a1f0-maas-api-openshift-ca" pod "4c493b7e-6578-4e59-bcd5-46e7e477a1f0" (UID: "4c493b7e-6578-4e59-bcd5-46e7e477a1f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:00:21.379436 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:21.379414    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f230091-46b6-4e43-9241-616506e54320-maas-api-service-ca" pod "7f230091-46b6-4e43-9241-616506e54320" (UID: "7f230091-46b6-4e43-9241-616506e54320"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:00:21.379831 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:21.379806    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/870683a9-9f9e-4071-a4e2-4e62a0178309-maas-api-service-ca" pod "870683a9-9f9e-4071-a4e2-4e62a0178309" (UID: "870683a9-9f9e-4071-a4e2-4e62a0178309"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:00:21.382809 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:21.382776    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c493b7e-6578-4e59-bcd5-46e7e477a1f0-maas-api-xks-ca" pod "4c493b7e-6578-4e59-bcd5-46e7e477a1f0" (UID: "4c493b7e-6578-4e59-bcd5-46e7e477a1f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:00:21.383575 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:21.383545    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/870683a9-9f9e-4071-a4e2-4e62a0178309-kube-api-access-xfmsg" pod "870683a9-9f9e-4071-a4e2-4e62a0178309" (UID: "870683a9-9f9e-4071-a4e2-4e62a0178309"). InnerVolumeSpecName "kube-api-access-xfmsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:00:21.383780 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:21.383728    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c493b7e-6578-4e59-bcd5-46e7e477a1f0-kube-api-access-hdqbq" pod "4c493b7e-6578-4e59-bcd5-46e7e477a1f0" (UID: "4c493b7e-6578-4e59-bcd5-46e7e477a1f0"). InnerVolumeSpecName "kube-api-access-hdqbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:00:21.393794 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:21.393753    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f230091-46b6-4e43-9241-616506e54320-kube-api-access-g5z2z" pod "7f230091-46b6-4e43-9241-616506e54320" (UID: "7f230091-46b6-4e43-9241-616506e54320"). InnerVolumeSpecName "kube-api-access-g5z2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:00:21.479343 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:21.479303    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5z2z\" (UniqueName: \"kubernetes.io/projected/7f230091-46b6-4e43-9241-616506e54320-kube-api-access-g5z2z\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:00:21.479343 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:21.479337    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/870683a9-9f9e-4071-a4e2-4e62a0178309-maas-api-service-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:00:21.479343 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:21.479347    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfmsg\" (UniqueName: \"kubernetes.io/projected/870683a9-9f9e-4071-a4e2-4e62a0178309-kube-api-access-xfmsg\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:00:21.479600 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:21.479357    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c493b7e-6578-4e59-bcd5-46e7e477a1f0-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:00:21.479600 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:21.479367    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c493b7e-6578-4e59-bcd5-46e7e477a1f0-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:00:21.479600 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:21.479375    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f230091-46b6-4e43-9241-616506e54320-maas-api-service-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:00:21.479600 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:21.479386    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdqbq\" (UniqueName: \"kubernetes.io/projected/4c493b7e-6578-4e59-bcd5-46e7e477a1f0-kube-api-access-hdqbq\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:00:21.608339 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:21.608306    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895480-zb2db"]
Nov 03 18:00:21.611726 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:21.611698    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895480-zb2db"]
Nov 03 18:00:22.109663 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:22.109633    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b3a56f-zf47h"
Nov 03 18:00:22.109880 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:22.109654    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b3a56f-zf47h" event={"ID":"4c493b7e-6578-4e59-bcd5-46e7e477a1f0","Type":"ContainerDied","Data":"ec12fb1773cf8b029d341fe2a8a643286815a0c600ece0a95699cabc580f80a3"}
Nov 03 18:00:22.109880 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:22.109712    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec12fb1773cf8b029d341fe2a8a643286815a0c600ece0a95699cabc580f80a3"
Nov 03 18:00:22.111122 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:22.111100    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0ccbb9-29895480-wg82z"
Nov 03 18:00:22.111226 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:22.111092    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0ccbb9-29895480-wg82z" event={"ID":"7f230091-46b6-4e43-9241-616506e54320","Type":"ContainerDied","Data":"b8a02d54956aa14233ffbd8d16a0251dad2fe96a494cc8809334f6486c32e3b4"}
Nov 03 18:00:22.111373 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:22.111228    5315 scope.go:122] "RemoveContainer" containerID="cad629f85d0e479807faee6e0213a3d0355814dd7b148caeb060974022561685"
Nov 03 18:00:22.112654 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:22.112632    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895480-v2xwh"
Nov 03 18:00:22.112845 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:22.112603    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895480-v2xwh" event={"ID":"870683a9-9f9e-4071-a4e2-4e62a0178309","Type":"ContainerDied","Data":"7f5241ac414186f0c349bd44f14b491467af9579aaaf2b44fff05bb2a9581986"}
Nov 03 18:00:22.121886 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:22.121859    5315 scope.go:122] "RemoveContainer" containerID="4969327b29d55c34b91c86665f96f6ede2198905f5d48229b3246ff97226f336"
Nov 03 18:00:22.267462 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:22.267426    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895480-v2xwh"]
Nov 03 18:00:22.270906 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:22.270863    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895480-v2xwh"]
Nov 03 18:00:22.423878 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:22.423787    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0ccbb9-29895480-wg82z"]
Nov 03 18:00:22.426104 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:22.426077    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0ccbb9-29895480-wg82z"]
Nov 03 18:00:23.104477 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:23.104444    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5026ac7e-119a-4c3d-865c-848d68f35bdf" path="/var/lib/kubelet/pods/5026ac7e-119a-4c3d-865c-848d68f35bdf/volumes"
Nov 03 18:00:23.104963 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:23.104853    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f230091-46b6-4e43-9241-616506e54320" path="/var/lib/kubelet/pods/7f230091-46b6-4e43-9241-616506e54320/volumes"
Nov 03 18:00:23.105191 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:23.105178    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="870683a9-9f9e-4071-a4e2-4e62a0178309" path="/var/lib/kubelet/pods/870683a9-9f9e-4071-a4e2-4e62a0178309/volumes"
Nov 03 18:00:23.105819 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:23.105804    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8c7d4f4-916e-4418-aa0a-2e7e844583ba" path="/var/lib/kubelet/pods/a8c7d4f4-916e-4418-aa0a-2e7e844583ba/volumes"
Nov 03 18:00:24.578370 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:00:24.578335    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:00:28.334590 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.334552    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fe07fd3-ngtvh"]
Nov 03 18:00:28.335133 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.335006    5315 cpu_manager.go:414] "containerMap: removing container" podUID="870683a9-9f9e-4071-a4e2-4e62a0178309" containerName="cleanup"
Nov 03 18:00:28.335133 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.335020    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="870683a9-9f9e-4071-a4e2-4e62a0178309" containerName="cleanup"
Nov 03 18:00:28.335133 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.335050    5315 cpu_manager.go:414] "containerMap: removing container" podUID="a8c7d4f4-916e-4418-aa0a-2e7e844583ba" containerName="cleanup"
Nov 03 18:00:28.335133 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.335055    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8c7d4f4-916e-4418-aa0a-2e7e844583ba" containerName="cleanup"
Nov 03 18:00:28.335133 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.335061    5315 cpu_manager.go:414] "containerMap: removing container" podUID="5026ac7e-119a-4c3d-865c-848d68f35bdf" containerName="cleanup"
Nov 03 18:00:28.335133 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.335065    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5026ac7e-119a-4c3d-865c-848d68f35bdf" containerName="cleanup"
Nov 03 18:00:28.335133 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.335073    5315 cpu_manager.go:414] "containerMap: removing container" podUID="a8c7d4f4-916e-4418-aa0a-2e7e844583ba" containerName="cleanup"
Nov 03 18:00:28.335133 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.335078    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8c7d4f4-916e-4418-aa0a-2e7e844583ba" containerName="cleanup"
Nov 03 18:00:28.335133 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.335091    5315 cpu_manager.go:414] "containerMap: removing container" podUID="7f230091-46b6-4e43-9241-616506e54320" containerName="cleanup"
Nov 03 18:00:28.335133 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.335095    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f230091-46b6-4e43-9241-616506e54320" containerName="cleanup"
Nov 03 18:00:28.335133 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.335105    5315 cpu_manager.go:414] "containerMap: removing container" podUID="944b5b5f-42cf-4c48-8676-b36b0b4dd49f" containerName="cleanup"
Nov 03 18:00:28.335133 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.335109    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="944b5b5f-42cf-4c48-8676-b36b0b4dd49f" containerName="cleanup"
Nov 03 18:00:28.335133 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.335116    5315 cpu_manager.go:414] "containerMap: removing container" podUID="7f230091-46b6-4e43-9241-616506e54320" containerName="cleanup"
Nov 03 18:00:28.335133 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.335120    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f230091-46b6-4e43-9241-616506e54320" containerName="cleanup"
Nov 03 18:00:28.335133 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.335126    5315 cpu_manager.go:414] "containerMap: removing container" podUID="5026ac7e-119a-4c3d-865c-848d68f35bdf" containerName="cleanup"
Nov 03 18:00:28.335133 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.335132    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5026ac7e-119a-4c3d-865c-848d68f35bdf" containerName="cleanup"
Nov 03 18:00:28.335133 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.335138    5315 cpu_manager.go:414] "containerMap: removing container" podUID="870683a9-9f9e-4071-a4e2-4e62a0178309" containerName="cleanup"
Nov 03 18:00:28.335133 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.335143    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="870683a9-9f9e-4071-a4e2-4e62a0178309" containerName="cleanup"
Nov 03 18:00:28.335996 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.335150    5315 cpu_manager.go:414] "containerMap: removing container" podUID="7f230091-46b6-4e43-9241-616506e54320" containerName="cleanup"
Nov 03 18:00:28.335996 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.335154    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f230091-46b6-4e43-9241-616506e54320" containerName="cleanup"
Nov 03 18:00:28.335996 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.335162    5315 cpu_manager.go:414] "containerMap: removing container" podUID="4c493b7e-6578-4e59-bcd5-46e7e477a1f0" containerName="revoke-keys"
Nov 03 18:00:28.335996 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.335167    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c493b7e-6578-4e59-bcd5-46e7e477a1f0" containerName="revoke-keys"
Nov 03 18:00:28.335996 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.335225    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8c7d4f4-916e-4418-aa0a-2e7e844583ba" containerName="cleanup"
Nov 03 18:00:28.335996 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.335231    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="5026ac7e-119a-4c3d-865c-848d68f35bdf" containerName="cleanup"
Nov 03 18:00:28.335996 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.335237    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="870683a9-9f9e-4071-a4e2-4e62a0178309" containerName="cleanup"
Nov 03 18:00:28.335996 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.335242    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8c7d4f4-916e-4418-aa0a-2e7e844583ba" containerName="cleanup"
Nov 03 18:00:28.335996 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.335249    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="5026ac7e-119a-4c3d-865c-848d68f35bdf" containerName="cleanup"
Nov 03 18:00:28.335996 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.335254    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f230091-46b6-4e43-9241-616506e54320" containerName="cleanup"
Nov 03 18:00:28.335996 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.335260    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f230091-46b6-4e43-9241-616506e54320" containerName="cleanup"
Nov 03 18:00:28.335996 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.335266    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f230091-46b6-4e43-9241-616506e54320" containerName="cleanup"
Nov 03 18:00:28.335996 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.335271    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="5026ac7e-119a-4c3d-865c-848d68f35bdf" containerName="cleanup"
Nov 03 18:00:28.335996 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.335275    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="870683a9-9f9e-4071-a4e2-4e62a0178309" containerName="cleanup"
Nov 03 18:00:28.335996 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.335280    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c493b7e-6578-4e59-bcd5-46e7e477a1f0" containerName="revoke-keys"
Nov 03 18:00:28.335996 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.335285    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="870683a9-9f9e-4071-a4e2-4e62a0178309" containerName="cleanup"
Nov 03 18:00:28.339956 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.339931    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe07fd3-ngtvh"
Nov 03 18:00:28.388630 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.388573    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:00:28.432537 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.432503    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cswws\" (UniqueName: \"kubernetes.io/projected/796733e8-258e-4378-b1d2-974722376ccd-kube-api-access-cswws\") pod \"maas-api-revoke-sub-1fe07fd3-ngtvh\" (UID: \"796733e8-258e-4378-b1d2-974722376ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe07fd3-ngtvh"
Nov 03 18:00:28.432733 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.432543    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/796733e8-258e-4378-b1d2-974722376ccd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fe07fd3-ngtvh\" (UID: \"796733e8-258e-4378-b1d2-974722376ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe07fd3-ngtvh"
Nov 03 18:00:28.432782 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.432741    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/796733e8-258e-4378-b1d2-974722376ccd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fe07fd3-ngtvh\" (UID: \"796733e8-258e-4378-b1d2-974722376ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe07fd3-ngtvh"
Nov 03 18:00:28.434501 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.434472    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:00:28.434604 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.434504    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:00:28.434604 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.434588    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:00:28.436668 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.436652    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fe07fd3-ngtvh"]
Nov 03 18:00:28.533412 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.533382    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/796733e8-258e-4378-b1d2-974722376ccd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fe07fd3-ngtvh\" (UID: \"796733e8-258e-4378-b1d2-974722376ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe07fd3-ngtvh"
Nov 03 18:00:28.533586 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.533430    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cswws\" (UniqueName: \"kubernetes.io/projected/796733e8-258e-4378-b1d2-974722376ccd-kube-api-access-cswws\") pod \"maas-api-revoke-sub-1fe07fd3-ngtvh\" (UID: \"796733e8-258e-4378-b1d2-974722376ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe07fd3-ngtvh"
Nov 03 18:00:28.533586 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.533555    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/796733e8-258e-4378-b1d2-974722376ccd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fe07fd3-ngtvh\" (UID: \"796733e8-258e-4378-b1d2-974722376ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe07fd3-ngtvh"
Nov 03 18:00:28.534216 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.534188    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/796733e8-258e-4378-b1d2-974722376ccd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fe07fd3-ngtvh\" (UID: \"796733e8-258e-4378-b1d2-974722376ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe07fd3-ngtvh"
Nov 03 18:00:28.535812 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.535790    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/796733e8-258e-4378-b1d2-974722376ccd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fe07fd3-ngtvh\" (UID: \"796733e8-258e-4378-b1d2-974722376ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe07fd3-ngtvh"
Nov 03 18:00:28.591891 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.591814    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cswws\" (UniqueName: \"kubernetes.io/projected/796733e8-258e-4378-b1d2-974722376ccd-kube-api-access-cswws\") pod \"maas-api-revoke-sub-1fe07fd3-ngtvh\" (UID: \"796733e8-258e-4378-b1d2-974722376ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe07fd3-ngtvh"
Nov 03 18:00:28.654760 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:28.654726    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe07fd3-ngtvh"
Nov 03 18:00:29.026890 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:29.026856    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fe07fd3-ngtvh"]
Nov 03 18:00:29.033728 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:00:29.033698    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod796733e8_258e_4378_b1d2_974722376ccd.slice/crio-5e53c5d59e3dbc218b3641afd4693df6290ad2ca2fb6a2f9600fd8ba13495c9d WatchSource:0}: Error finding container 5e53c5d59e3dbc218b3641afd4693df6290ad2ca2fb6a2f9600fd8ba13495c9d: Status 404 returned error can't find the container with id 5e53c5d59e3dbc218b3641afd4693df6290ad2ca2fb6a2f9600fd8ba13495c9d
Nov 03 18:00:29.155693 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:29.155657    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe07fd3-ngtvh" event={"ID":"796733e8-258e-4378-b1d2-974722376ccd","Type":"ContainerStarted","Data":"5e53c5d59e3dbc218b3641afd4693df6290ad2ca2fb6a2f9600fd8ba13495c9d"}
Nov 03 18:00:30.162494 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:30.162396    5315 generic.go:356] "Generic (PLEG): container finished" podID="796733e8-258e-4378-b1d2-974722376ccd" containerID="c8058cd11709e2bd49974f2d27a0b154b980ac3032b554f318a70174faed6480" exitCode=0
Nov 03 18:00:30.162971 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:30.162482    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe07fd3-ngtvh" event={"ID":"796733e8-258e-4378-b1d2-974722376ccd","Type":"ContainerDied","Data":"c8058cd11709e2bd49974f2d27a0b154b980ac3032b554f318a70174faed6480"}
Nov 03 18:00:31.309976 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:31.309947    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe07fd3-ngtvh"
Nov 03 18:00:31.361883 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:31.361840    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/796733e8-258e-4378-b1d2-974722376ccd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/796733e8-258e-4378-b1d2-974722376ccd-maas-api-openshift-ca\") pod \"796733e8-258e-4378-b1d2-974722376ccd\" (UID: \"796733e8-258e-4378-b1d2-974722376ccd\") "
Nov 03 18:00:31.362060 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:31.361941    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/796733e8-258e-4378-b1d2-974722376ccd-kube-api-access-cswws\" (UniqueName: \"kubernetes.io/projected/796733e8-258e-4378-b1d2-974722376ccd-kube-api-access-cswws\") pod \"796733e8-258e-4378-b1d2-974722376ccd\" (UID: \"796733e8-258e-4378-b1d2-974722376ccd\") "
Nov 03 18:00:31.362060 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:31.361971    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/796733e8-258e-4378-b1d2-974722376ccd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/796733e8-258e-4378-b1d2-974722376ccd-maas-api-xks-ca\") pod \"796733e8-258e-4378-b1d2-974722376ccd\" (UID: \"796733e8-258e-4378-b1d2-974722376ccd\") "
Nov 03 18:00:31.362197 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:31.362170    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/796733e8-258e-4378-b1d2-974722376ccd-maas-api-openshift-ca" pod "796733e8-258e-4378-b1d2-974722376ccd" (UID: "796733e8-258e-4378-b1d2-974722376ccd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:00:31.364090 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:31.364069    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/796733e8-258e-4378-b1d2-974722376ccd-kube-api-access-cswws" pod "796733e8-258e-4378-b1d2-974722376ccd" (UID: "796733e8-258e-4378-b1d2-974722376ccd"). InnerVolumeSpecName "kube-api-access-cswws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:00:31.364150 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:31.364120    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/796733e8-258e-4378-b1d2-974722376ccd-maas-api-xks-ca" pod "796733e8-258e-4378-b1d2-974722376ccd" (UID: "796733e8-258e-4378-b1d2-974722376ccd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:00:31.463145 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:31.463068    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cswws\" (UniqueName: \"kubernetes.io/projected/796733e8-258e-4378-b1d2-974722376ccd-kube-api-access-cswws\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:00:31.463145 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:31.463103    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/796733e8-258e-4378-b1d2-974722376ccd-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:00:31.463145 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:31.463114    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/796733e8-258e-4378-b1d2-974722376ccd-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:00:32.174814 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:32.174774    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe07fd3-ngtvh" event={"ID":"796733e8-258e-4378-b1d2-974722376ccd","Type":"ContainerDied","Data":"5e53c5d59e3dbc218b3641afd4693df6290ad2ca2fb6a2f9600fd8ba13495c9d"}
Nov 03 18:00:32.174814 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:32.174799    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe07fd3-ngtvh"
Nov 03 18:00:32.175029 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:32.174834    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e53c5d59e3dbc218b3641afd4693df6290ad2ca2fb6a2f9600fd8ba13495c9d"
Nov 03 18:00:45.767224 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:45.767190    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f298b25a-pjz76"]
Nov 03 18:00:45.767647 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:45.767599    5315 cpu_manager.go:414] "containerMap: removing container" podUID="870683a9-9f9e-4071-a4e2-4e62a0178309" containerName="cleanup"
Nov 03 18:00:45.767647 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:45.767622    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="870683a9-9f9e-4071-a4e2-4e62a0178309" containerName="cleanup"
Nov 03 18:00:45.767735 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:45.767650    5315 cpu_manager.go:414] "containerMap: removing container" podUID="796733e8-258e-4378-b1d2-974722376ccd" containerName="revoke-keys"
Nov 03 18:00:45.767735 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:45.767655    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="796733e8-258e-4378-b1d2-974722376ccd" containerName="revoke-keys"
Nov 03 18:00:45.767735 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:45.767663    5315 cpu_manager.go:414] "containerMap: removing container" podUID="a8c7d4f4-916e-4418-aa0a-2e7e844583ba" containerName="cleanup"
Nov 03 18:00:45.767735 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:45.767668    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8c7d4f4-916e-4418-aa0a-2e7e844583ba" containerName="cleanup"
Nov 03 18:00:45.767735 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:45.767684    5315 cpu_manager.go:414] "containerMap: removing container" podUID="5026ac7e-119a-4c3d-865c-848d68f35bdf" containerName="cleanup"
Nov 03 18:00:45.767735 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:45.767689    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5026ac7e-119a-4c3d-865c-848d68f35bdf" containerName="cleanup"
Nov 03 18:00:45.767908 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:45.767747    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="796733e8-258e-4378-b1d2-974722376ccd" containerName="revoke-keys"
Nov 03 18:00:45.767908 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:45.767756    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8c7d4f4-916e-4418-aa0a-2e7e844583ba" containerName="cleanup"
Nov 03 18:00:45.772927 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:45.772903    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f298b25a-pjz76"
Nov 03 18:00:45.818152 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:45.818108    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e8bb82a-0c44-491b-a255-ee8e5c925ab2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f298b25a-pjz76\" (UID: \"7e8bb82a-0c44-491b-a255-ee8e5c925ab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f298b25a-pjz76"
Nov 03 18:00:45.818315 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:45.818176    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjrt5\" (UniqueName: \"kubernetes.io/projected/7e8bb82a-0c44-491b-a255-ee8e5c925ab2-kube-api-access-cjrt5\") pod \"maas-api-revoke-sub-f298b25a-pjz76\" (UID: \"7e8bb82a-0c44-491b-a255-ee8e5c925ab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f298b25a-pjz76"
Nov 03 18:00:45.818315 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:45.818257    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e8bb82a-0c44-491b-a255-ee8e5c925ab2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f298b25a-pjz76\" (UID: \"7e8bb82a-0c44-491b-a255-ee8e5c925ab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f298b25a-pjz76"
Nov 03 18:00:45.821305 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:45.821286    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:00:45.867346 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:45.867300    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:00:45.867346 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:45.867358    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:00:45.867592 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:45.867398    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:00:45.869471 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:45.869452    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f298b25a-pjz76"]
Nov 03 18:00:45.919531 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:45.919501    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e8bb82a-0c44-491b-a255-ee8e5c925ab2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f298b25a-pjz76\" (UID: \"7e8bb82a-0c44-491b-a255-ee8e5c925ab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f298b25a-pjz76"
Nov 03 18:00:45.919731 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:45.919553    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjrt5\" (UniqueName: \"kubernetes.io/projected/7e8bb82a-0c44-491b-a255-ee8e5c925ab2-kube-api-access-cjrt5\") pod \"maas-api-revoke-sub-f298b25a-pjz76\" (UID: \"7e8bb82a-0c44-491b-a255-ee8e5c925ab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f298b25a-pjz76"
Nov 03 18:00:45.919731 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:45.919579    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e8bb82a-0c44-491b-a255-ee8e5c925ab2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f298b25a-pjz76\" (UID: \"7e8bb82a-0c44-491b-a255-ee8e5c925ab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f298b25a-pjz76"
Nov 03 18:00:45.920165 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:45.920144    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e8bb82a-0c44-491b-a255-ee8e5c925ab2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f298b25a-pjz76\" (UID: \"7e8bb82a-0c44-491b-a255-ee8e5c925ab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f298b25a-pjz76"
Nov 03 18:00:45.921939 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:45.921916    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e8bb82a-0c44-491b-a255-ee8e5c925ab2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f298b25a-pjz76\" (UID: \"7e8bb82a-0c44-491b-a255-ee8e5c925ab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f298b25a-pjz76"
Nov 03 18:00:45.977054 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:45.977028    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjrt5\" (UniqueName: \"kubernetes.io/projected/7e8bb82a-0c44-491b-a255-ee8e5c925ab2-kube-api-access-cjrt5\") pod \"maas-api-revoke-sub-f298b25a-pjz76\" (UID: \"7e8bb82a-0c44-491b-a255-ee8e5c925ab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f298b25a-pjz76"
Nov 03 18:00:46.086577 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:46.086545    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f298b25a-pjz76"
Nov 03 18:00:46.460509 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:46.460421    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f298b25a-pjz76"]
Nov 03 18:00:46.467662 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:00:46.467603    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e8bb82a_0c44_491b_a255_ee8e5c925ab2.slice/crio-fa377c82ef0492d83d7503e33a2c9f264ca0d5f9195ebc0deffbdd8e71eed1cf WatchSource:0}: Error finding container fa377c82ef0492d83d7503e33a2c9f264ca0d5f9195ebc0deffbdd8e71eed1cf: Status 404 returned error can't find the container with id fa377c82ef0492d83d7503e33a2c9f264ca0d5f9195ebc0deffbdd8e71eed1cf
Nov 03 18:00:47.274005 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:47.273950    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f298b25a-pjz76" event={"ID":"7e8bb82a-0c44-491b-a255-ee8e5c925ab2","Type":"ContainerStarted","Data":"5b9be3dfca82b13962ed31c7dc7f54bc790b899bf8f901c27fe9f769930d450b"}
Nov 03 18:00:47.274453 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:47.274034    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f298b25a-pjz76" event={"ID":"7e8bb82a-0c44-491b-a255-ee8e5c925ab2","Type":"ContainerStarted","Data":"fa377c82ef0492d83d7503e33a2c9f264ca0d5f9195ebc0deffbdd8e71eed1cf"}
Nov 03 18:00:47.386576 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:47.386522    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f298b25a-pjz76" podStartSLOduration=2.386507826 podStartE2EDuration="2.386507826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:00:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:00:47.385127968 +0000 UTC m=+886.401326036" watchObservedRunningTime="2026-11-03 18:00:47.386507826 +0000 UTC m=+886.402705934"
Nov 03 18:00:48.281567 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:48.281534    5315 generic.go:356] "Generic (PLEG): container finished" podID="7e8bb82a-0c44-491b-a255-ee8e5c925ab2" containerID="5b9be3dfca82b13962ed31c7dc7f54bc790b899bf8f901c27fe9f769930d450b" exitCode=0
Nov 03 18:00:48.282017 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:48.281635    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f298b25a-pjz76" event={"ID":"7e8bb82a-0c44-491b-a255-ee8e5c925ab2","Type":"ContainerDied","Data":"5b9be3dfca82b13962ed31c7dc7f54bc790b899bf8f901c27fe9f769930d450b"}
Nov 03 18:00:49.439849 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:49.439824    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f298b25a-pjz76"
Nov 03 18:00:49.560409 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:49.560363    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e8bb82a-0c44-491b-a255-ee8e5c925ab2-kube-api-access-cjrt5\" (UniqueName: \"kubernetes.io/projected/7e8bb82a-0c44-491b-a255-ee8e5c925ab2-kube-api-access-cjrt5\") pod \"7e8bb82a-0c44-491b-a255-ee8e5c925ab2\" (UID: \"7e8bb82a-0c44-491b-a255-ee8e5c925ab2\") "
Nov 03 18:00:49.560596 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:49.560455    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e8bb82a-0c44-491b-a255-ee8e5c925ab2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e8bb82a-0c44-491b-a255-ee8e5c925ab2-maas-api-openshift-ca\") pod \"7e8bb82a-0c44-491b-a255-ee8e5c925ab2\" (UID: \"7e8bb82a-0c44-491b-a255-ee8e5c925ab2\") "
Nov 03 18:00:49.560740 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:49.560636    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e8bb82a-0c44-491b-a255-ee8e5c925ab2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e8bb82a-0c44-491b-a255-ee8e5c925ab2-maas-api-xks-ca\") pod \"7e8bb82a-0c44-491b-a255-ee8e5c925ab2\" (UID: \"7e8bb82a-0c44-491b-a255-ee8e5c925ab2\") "
Nov 03 18:00:49.560868 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:49.560838    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e8bb82a-0c44-491b-a255-ee8e5c925ab2-maas-api-openshift-ca" pod "7e8bb82a-0c44-491b-a255-ee8e5c925ab2" (UID: "7e8bb82a-0c44-491b-a255-ee8e5c925ab2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:00:49.561045 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:49.561022    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e8bb82a-0c44-491b-a255-ee8e5c925ab2-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:00:49.563075 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:49.563041    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e8bb82a-0c44-491b-a255-ee8e5c925ab2-maas-api-xks-ca" pod "7e8bb82a-0c44-491b-a255-ee8e5c925ab2" (UID: "7e8bb82a-0c44-491b-a255-ee8e5c925ab2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:00:49.563196 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:49.563084    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e8bb82a-0c44-491b-a255-ee8e5c925ab2-kube-api-access-cjrt5" pod "7e8bb82a-0c44-491b-a255-ee8e5c925ab2" (UID: "7e8bb82a-0c44-491b-a255-ee8e5c925ab2"). InnerVolumeSpecName "kube-api-access-cjrt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:00:49.661755 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:49.661714    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjrt5\" (UniqueName: \"kubernetes.io/projected/7e8bb82a-0c44-491b-a255-ee8e5c925ab2-kube-api-access-cjrt5\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:00:49.661755 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:49.661747    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e8bb82a-0c44-491b-a255-ee8e5c925ab2-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:00:50.296171 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:50.296137    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f298b25a-pjz76"
Nov 03 18:00:50.296171 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:50.296152    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f298b25a-pjz76" event={"ID":"7e8bb82a-0c44-491b-a255-ee8e5c925ab2","Type":"ContainerDied","Data":"fa377c82ef0492d83d7503e33a2c9f264ca0d5f9195ebc0deffbdd8e71eed1cf"}
Nov 03 18:00:50.296421 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:50.296201    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa377c82ef0492d83d7503e33a2c9f264ca0d5f9195ebc0deffbdd8e71eed1cf"
Nov 03 18:00:50.706538 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:50.706454    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b70309da-gtm7j"]
Nov 03 18:00:50.706950 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:50.706934    5315 cpu_manager.go:414] "containerMap: removing container" podUID="7e8bb82a-0c44-491b-a255-ee8e5c925ab2" containerName="revoke-keys"
Nov 03 18:00:50.706992 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:50.706951    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e8bb82a-0c44-491b-a255-ee8e5c925ab2" containerName="revoke-keys"
Nov 03 18:00:50.707045 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:50.707036    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e8bb82a-0c44-491b-a255-ee8e5c925ab2" containerName="revoke-keys"
Nov 03 18:00:50.710463 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:50.710437    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b70309da-gtm7j"
Nov 03 18:00:50.774941 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:50.774906    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b17a20a-d840-4bc2-a343-e84fea40beb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b70309da-gtm7j\" (UID: \"5b17a20a-d840-4bc2-a343-e84fea40beb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b70309da-gtm7j"
Nov 03 18:00:50.775121 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:50.774950    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgpqv\" (UniqueName: \"kubernetes.io/projected/5b17a20a-d840-4bc2-a343-e84fea40beb5-kube-api-access-kgpqv\") pod \"maas-api-revoke-sub-b70309da-gtm7j\" (UID: \"5b17a20a-d840-4bc2-a343-e84fea40beb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b70309da-gtm7j"
Nov 03 18:00:50.775121 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:50.774993    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b17a20a-d840-4bc2-a343-e84fea40beb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b70309da-gtm7j\" (UID: \"5b17a20a-d840-4bc2-a343-e84fea40beb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b70309da-gtm7j"
Nov 03 18:00:50.805252 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:50.805202    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:00:50.805252 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:50.805229    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:00:50.805252 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:50.805209    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:00:50.805553 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:50.805262    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:00:50.807512 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:50.807486    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b70309da-gtm7j"]
Nov 03 18:00:50.876447 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:50.876409    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b17a20a-d840-4bc2-a343-e84fea40beb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b70309da-gtm7j\" (UID: \"5b17a20a-d840-4bc2-a343-e84fea40beb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b70309da-gtm7j"
Nov 03 18:00:50.876447 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:50.876448    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgpqv\" (UniqueName: \"kubernetes.io/projected/5b17a20a-d840-4bc2-a343-e84fea40beb5-kube-api-access-kgpqv\") pod \"maas-api-revoke-sub-b70309da-gtm7j\" (UID: \"5b17a20a-d840-4bc2-a343-e84fea40beb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b70309da-gtm7j"
Nov 03 18:00:50.876666 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:50.876483    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b17a20a-d840-4bc2-a343-e84fea40beb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b70309da-gtm7j\" (UID: \"5b17a20a-d840-4bc2-a343-e84fea40beb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b70309da-gtm7j"
Nov 03 18:00:50.877109 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:50.877087    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b17a20a-d840-4bc2-a343-e84fea40beb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b70309da-gtm7j\" (UID: \"5b17a20a-d840-4bc2-a343-e84fea40beb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b70309da-gtm7j"
Nov 03 18:00:50.878828 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:50.878803    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b17a20a-d840-4bc2-a343-e84fea40beb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b70309da-gtm7j\" (UID: \"5b17a20a-d840-4bc2-a343-e84fea40beb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b70309da-gtm7j"
Nov 03 18:00:50.934011 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:50.933978    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgpqv\" (UniqueName: \"kubernetes.io/projected/5b17a20a-d840-4bc2-a343-e84fea40beb5-kube-api-access-kgpqv\") pod \"maas-api-revoke-sub-b70309da-gtm7j\" (UID: \"5b17a20a-d840-4bc2-a343-e84fea40beb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b70309da-gtm7j"
Nov 03 18:00:51.025092 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:51.025001    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b70309da-gtm7j"
Nov 03 18:00:51.402622 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:51.402568    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b70309da-gtm7j"]
Nov 03 18:00:51.408802 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:00:51.408761    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b17a20a_d840_4bc2_a343_e84fea40beb5.slice/crio-5147bdba298f70f3e146016e29c94a8a554ba1c333cd00d37a63dd35e028ccbf WatchSource:0}: Error finding container 5147bdba298f70f3e146016e29c94a8a554ba1c333cd00d37a63dd35e028ccbf: Status 404 returned error can't find the container with id 5147bdba298f70f3e146016e29c94a8a554ba1c333cd00d37a63dd35e028ccbf
Nov 03 18:00:52.308675 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:52.308633    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b70309da-gtm7j" event={"ID":"5b17a20a-d840-4bc2-a343-e84fea40beb5","Type":"ContainerStarted","Data":"52718e14e23520673d90530aed297f1a06ed79ef2b95d32514e05261def7b512"}
Nov 03 18:00:52.309170 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:52.308695    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b70309da-gtm7j" event={"ID":"5b17a20a-d840-4bc2-a343-e84fea40beb5","Type":"ContainerStarted","Data":"5147bdba298f70f3e146016e29c94a8a554ba1c333cd00d37a63dd35e028ccbf"}
Nov 03 18:00:52.420294 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:52.420238    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b70309da-gtm7j" podStartSLOduration=2.420223419 podStartE2EDuration="2.420223419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:00:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:00:52.419120002 +0000 UTC m=+891.435318071" watchObservedRunningTime="2026-11-03 18:00:52.420223419 +0000 UTC m=+891.436421488"
Nov 03 18:00:52.568247 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:52.568158    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c73d193e-svwzg"]
Nov 03 18:00:52.571840 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:52.571818    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c73d193e-svwzg"
Nov 03 18:00:52.634941 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:52.634909    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c73d193e-svwzg"]
Nov 03 18:00:52.694207 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:52.694171    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98ffe06e-3793-47ae-b331-b29dd02eaa4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c73d193e-svwzg\" (UID: \"98ffe06e-3793-47ae-b331-b29dd02eaa4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c73d193e-svwzg"
Nov 03 18:00:52.694417 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:52.694389    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxsdq\" (UniqueName: \"kubernetes.io/projected/98ffe06e-3793-47ae-b331-b29dd02eaa4e-kube-api-access-kxsdq\") pod \"maas-api-revoke-sub-c73d193e-svwzg\" (UID: \"98ffe06e-3793-47ae-b331-b29dd02eaa4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c73d193e-svwzg"
Nov 03 18:00:52.694529 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:52.694506    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98ffe06e-3793-47ae-b331-b29dd02eaa4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c73d193e-svwzg\" (UID: \"98ffe06e-3793-47ae-b331-b29dd02eaa4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c73d193e-svwzg"
Nov 03 18:00:52.795823 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:52.795783    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxsdq\" (UniqueName: \"kubernetes.io/projected/98ffe06e-3793-47ae-b331-b29dd02eaa4e-kube-api-access-kxsdq\") pod \"maas-api-revoke-sub-c73d193e-svwzg\" (UID: \"98ffe06e-3793-47ae-b331-b29dd02eaa4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c73d193e-svwzg"
Nov 03 18:00:52.796004 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:52.795840    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98ffe06e-3793-47ae-b331-b29dd02eaa4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c73d193e-svwzg\" (UID: \"98ffe06e-3793-47ae-b331-b29dd02eaa4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c73d193e-svwzg"
Nov 03 18:00:52.796004 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:52.795877    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98ffe06e-3793-47ae-b331-b29dd02eaa4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c73d193e-svwzg\" (UID: \"98ffe06e-3793-47ae-b331-b29dd02eaa4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c73d193e-svwzg"
Nov 03 18:00:52.796600 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:52.796579    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98ffe06e-3793-47ae-b331-b29dd02eaa4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c73d193e-svwzg\" (UID: \"98ffe06e-3793-47ae-b331-b29dd02eaa4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c73d193e-svwzg"
Nov 03 18:00:52.798647 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:52.798622    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98ffe06e-3793-47ae-b331-b29dd02eaa4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c73d193e-svwzg\" (UID: \"98ffe06e-3793-47ae-b331-b29dd02eaa4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c73d193e-svwzg"
Nov 03 18:00:52.854178 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:52.854110    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxsdq\" (UniqueName: \"kubernetes.io/projected/98ffe06e-3793-47ae-b331-b29dd02eaa4e-kube-api-access-kxsdq\") pod \"maas-api-revoke-sub-c73d193e-svwzg\" (UID: \"98ffe06e-3793-47ae-b331-b29dd02eaa4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c73d193e-svwzg"
Nov 03 18:00:52.884297 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:52.884261    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c73d193e-svwzg"
Nov 03 18:00:53.259656 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:53.259561    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c73d193e-svwzg"]
Nov 03 18:00:53.265714 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:00:53.265684    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98ffe06e_3793_47ae_b331_b29dd02eaa4e.slice/crio-6db7dd48cc4dae11c677186272a77f0fe30059c135c5814979316846fb4bbfed WatchSource:0}: Error finding container 6db7dd48cc4dae11c677186272a77f0fe30059c135c5814979316846fb4bbfed: Status 404 returned error can't find the container with id 6db7dd48cc4dae11c677186272a77f0fe30059c135c5814979316846fb4bbfed
Nov 03 18:00:53.315652 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:53.315599    5315 generic.go:356] "Generic (PLEG): container finished" podID="5b17a20a-d840-4bc2-a343-e84fea40beb5" containerID="52718e14e23520673d90530aed297f1a06ed79ef2b95d32514e05261def7b512" exitCode=0
Nov 03 18:00:53.316094 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:53.315681    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b70309da-gtm7j" event={"ID":"5b17a20a-d840-4bc2-a343-e84fea40beb5","Type":"ContainerDied","Data":"52718e14e23520673d90530aed297f1a06ed79ef2b95d32514e05261def7b512"}
Nov 03 18:00:53.316994 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:53.316971    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c73d193e-svwzg" event={"ID":"98ffe06e-3793-47ae-b331-b29dd02eaa4e","Type":"ContainerStarted","Data":"6db7dd48cc4dae11c677186272a77f0fe30059c135c5814979316846fb4bbfed"}
Nov 03 18:00:54.323544 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:54.323511    5315 generic.go:356] "Generic (PLEG): container finished" podID="98ffe06e-3793-47ae-b331-b29dd02eaa4e" containerID="f7c1e246d4a71c9958f4e2b48dc4129e1d606f4d86ab9daf1829a87bb6594594" exitCode=0
Nov 03 18:00:54.324035 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:54.323592    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c73d193e-svwzg" event={"ID":"98ffe06e-3793-47ae-b331-b29dd02eaa4e","Type":"ContainerDied","Data":"f7c1e246d4a71c9958f4e2b48dc4129e1d606f4d86ab9daf1829a87bb6594594"}
Nov 03 18:00:54.462003 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:54.461978    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b70309da-gtm7j"
Nov 03 18:00:54.512271 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:54.512238    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b17a20a-d840-4bc2-a343-e84fea40beb5-kube-api-access-kgpqv\" (UniqueName: \"kubernetes.io/projected/5b17a20a-d840-4bc2-a343-e84fea40beb5-kube-api-access-kgpqv\") pod \"5b17a20a-d840-4bc2-a343-e84fea40beb5\" (UID: \"5b17a20a-d840-4bc2-a343-e84fea40beb5\") "
Nov 03 18:00:54.512449 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:54.512335    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b17a20a-d840-4bc2-a343-e84fea40beb5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b17a20a-d840-4bc2-a343-e84fea40beb5-maas-api-openshift-ca\") pod \"5b17a20a-d840-4bc2-a343-e84fea40beb5\" (UID: \"5b17a20a-d840-4bc2-a343-e84fea40beb5\") "
Nov 03 18:00:54.512449 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:54.512360    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b17a20a-d840-4bc2-a343-e84fea40beb5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b17a20a-d840-4bc2-a343-e84fea40beb5-maas-api-xks-ca\") pod \"5b17a20a-d840-4bc2-a343-e84fea40beb5\" (UID: \"5b17a20a-d840-4bc2-a343-e84fea40beb5\") "
Nov 03 18:00:54.512717 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:54.512692    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b17a20a-d840-4bc2-a343-e84fea40beb5-maas-api-openshift-ca" pod "5b17a20a-d840-4bc2-a343-e84fea40beb5" (UID: "5b17a20a-d840-4bc2-a343-e84fea40beb5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:00:54.514474 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:54.514437    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b17a20a-d840-4bc2-a343-e84fea40beb5-kube-api-access-kgpqv" pod "5b17a20a-d840-4bc2-a343-e84fea40beb5" (UID: "5b17a20a-d840-4bc2-a343-e84fea40beb5"). InnerVolumeSpecName "kube-api-access-kgpqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:00:54.514474 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:54.514454    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b17a20a-d840-4bc2-a343-e84fea40beb5-maas-api-xks-ca" pod "5b17a20a-d840-4bc2-a343-e84fea40beb5" (UID: "5b17a20a-d840-4bc2-a343-e84fea40beb5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:00:54.578772 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:00:54.578732    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:00:54.614408 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:54.614373    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b17a20a-d840-4bc2-a343-e84fea40beb5-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:00:54.614408 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:54.614405    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgpqv\" (UniqueName: \"kubernetes.io/projected/5b17a20a-d840-4bc2-a343-e84fea40beb5-kube-api-access-kgpqv\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:00:54.614660 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:54.614420    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b17a20a-d840-4bc2-a343-e84fea40beb5-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:00:55.330527 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:55.330491    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b70309da-gtm7j" event={"ID":"5b17a20a-d840-4bc2-a343-e84fea40beb5","Type":"ContainerDied","Data":"5147bdba298f70f3e146016e29c94a8a554ba1c333cd00d37a63dd35e028ccbf"}
Nov 03 18:00:55.330983 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:55.330542    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5147bdba298f70f3e146016e29c94a8a554ba1c333cd00d37a63dd35e028ccbf"
Nov 03 18:00:55.330983 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:55.330643    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b70309da-gtm7j"
Nov 03 18:00:55.470140 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:55.470106    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c73d193e-svwzg"
Nov 03 18:00:55.524966 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:55.524933    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98ffe06e-3793-47ae-b331-b29dd02eaa4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98ffe06e-3793-47ae-b331-b29dd02eaa4e-maas-api-openshift-ca\") pod \"98ffe06e-3793-47ae-b331-b29dd02eaa4e\" (UID: \"98ffe06e-3793-47ae-b331-b29dd02eaa4e\") "
Nov 03 18:00:55.525125 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:55.525019    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98ffe06e-3793-47ae-b331-b29dd02eaa4e-kube-api-access-kxsdq\" (UniqueName: \"kubernetes.io/projected/98ffe06e-3793-47ae-b331-b29dd02eaa4e-kube-api-access-kxsdq\") pod \"98ffe06e-3793-47ae-b331-b29dd02eaa4e\" (UID: \"98ffe06e-3793-47ae-b331-b29dd02eaa4e\") "
Nov 03 18:00:55.525180 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:55.525131    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98ffe06e-3793-47ae-b331-b29dd02eaa4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98ffe06e-3793-47ae-b331-b29dd02eaa4e-maas-api-xks-ca\") pod \"98ffe06e-3793-47ae-b331-b29dd02eaa4e\" (UID: \"98ffe06e-3793-47ae-b331-b29dd02eaa4e\") "
Nov 03 18:00:55.525350 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:55.525328    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98ffe06e-3793-47ae-b331-b29dd02eaa4e-maas-api-openshift-ca" pod "98ffe06e-3793-47ae-b331-b29dd02eaa4e" (UID: "98ffe06e-3793-47ae-b331-b29dd02eaa4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:00:55.525508 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:55.525486    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98ffe06e-3793-47ae-b331-b29dd02eaa4e-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:00:55.527075 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:55.527044    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98ffe06e-3793-47ae-b331-b29dd02eaa4e-maas-api-xks-ca" pod "98ffe06e-3793-47ae-b331-b29dd02eaa4e" (UID: "98ffe06e-3793-47ae-b331-b29dd02eaa4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:00:55.527158 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:55.527135    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98ffe06e-3793-47ae-b331-b29dd02eaa4e-kube-api-access-kxsdq" pod "98ffe06e-3793-47ae-b331-b29dd02eaa4e" (UID: "98ffe06e-3793-47ae-b331-b29dd02eaa4e"). InnerVolumeSpecName "kube-api-access-kxsdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:00:55.626015 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:55.625980    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98ffe06e-3793-47ae-b331-b29dd02eaa4e-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:00:55.626015 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:55.626011    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxsdq\" (UniqueName: \"kubernetes.io/projected/98ffe06e-3793-47ae-b331-b29dd02eaa4e-kube-api-access-kxsdq\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:00:56.338636 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:56.338587    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c73d193e-svwzg" event={"ID":"98ffe06e-3793-47ae-b331-b29dd02eaa4e","Type":"ContainerDied","Data":"6db7dd48cc4dae11c677186272a77f0fe30059c135c5814979316846fb4bbfed"}
Nov 03 18:00:56.339201 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:56.339180    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6db7dd48cc4dae11c677186272a77f0fe30059c135c5814979316846fb4bbfed"
Nov 03 18:00:56.339303 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:56.338630    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c73d193e-svwzg"
Nov 03 18:00:57.953955 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:57.953920    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d421e35-kq2q2"]
Nov 03 18:00:57.954366 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:57.954352    5315 cpu_manager.go:414] "containerMap: removing container" podUID="5b17a20a-d840-4bc2-a343-e84fea40beb5" containerName="revoke-keys"
Nov 03 18:00:57.954434 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:57.954367    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b17a20a-d840-4bc2-a343-e84fea40beb5" containerName="revoke-keys"
Nov 03 18:00:57.954434 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:57.954388    5315 cpu_manager.go:414] "containerMap: removing container" podUID="98ffe06e-3793-47ae-b331-b29dd02eaa4e" containerName="revoke-keys"
Nov 03 18:00:57.954434 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:57.954393    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98ffe06e-3793-47ae-b331-b29dd02eaa4e" containerName="revoke-keys"
Nov 03 18:00:57.954542 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:57.954460    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b17a20a-d840-4bc2-a343-e84fea40beb5" containerName="revoke-keys"
Nov 03 18:00:57.954542 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:57.954471    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="98ffe06e-3793-47ae-b331-b29dd02eaa4e" containerName="revoke-keys"
Nov 03 18:00:57.958045 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:57.958023    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d421e35-kq2q2"
Nov 03 18:00:58.013516 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:58.013483    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:00:58.013727 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:58.013563    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:00:58.055590 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:58.055547    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x47lj\" (UniqueName: \"kubernetes.io/projected/ea648b10-b2ec-49ec-8b3d-cc10668990b1-kube-api-access-x47lj\") pod \"maas-api-revoke-sub-8d421e35-kq2q2\" (UID: \"ea648b10-b2ec-49ec-8b3d-cc10668990b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d421e35-kq2q2"
Nov 03 18:00:58.055780 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:58.055716    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea648b10-b2ec-49ec-8b3d-cc10668990b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d421e35-kq2q2\" (UID: \"ea648b10-b2ec-49ec-8b3d-cc10668990b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d421e35-kq2q2"
Nov 03 18:00:58.055780 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:58.055767    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea648b10-b2ec-49ec-8b3d-cc10668990b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d421e35-kq2q2\" (UID: \"ea648b10-b2ec-49ec-8b3d-cc10668990b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d421e35-kq2q2"
Nov 03 18:00:58.059483 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:58.059452    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:00:58.059709 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:58.059469    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:00:58.062066 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:58.062040    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d421e35-kq2q2"]
Nov 03 18:00:58.156866 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:58.156817    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x47lj\" (UniqueName: \"kubernetes.io/projected/ea648b10-b2ec-49ec-8b3d-cc10668990b1-kube-api-access-x47lj\") pod \"maas-api-revoke-sub-8d421e35-kq2q2\" (UID: \"ea648b10-b2ec-49ec-8b3d-cc10668990b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d421e35-kq2q2"
Nov 03 18:00:58.157046 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:58.156914    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea648b10-b2ec-49ec-8b3d-cc10668990b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d421e35-kq2q2\" (UID: \"ea648b10-b2ec-49ec-8b3d-cc10668990b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d421e35-kq2q2"
Nov 03 18:00:58.157046 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:58.156973    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea648b10-b2ec-49ec-8b3d-cc10668990b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d421e35-kq2q2\" (UID: \"ea648b10-b2ec-49ec-8b3d-cc10668990b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d421e35-kq2q2"
Nov 03 18:00:58.157727 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:58.157697    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea648b10-b2ec-49ec-8b3d-cc10668990b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d421e35-kq2q2\" (UID: \"ea648b10-b2ec-49ec-8b3d-cc10668990b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d421e35-kq2q2"
Nov 03 18:00:58.159462 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:58.159438    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea648b10-b2ec-49ec-8b3d-cc10668990b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d421e35-kq2q2\" (UID: \"ea648b10-b2ec-49ec-8b3d-cc10668990b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d421e35-kq2q2"
Nov 03 18:00:58.224174 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:58.224096    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x47lj\" (UniqueName: \"kubernetes.io/projected/ea648b10-b2ec-49ec-8b3d-cc10668990b1-kube-api-access-x47lj\") pod \"maas-api-revoke-sub-8d421e35-kq2q2\" (UID: \"ea648b10-b2ec-49ec-8b3d-cc10668990b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d421e35-kq2q2"
Nov 03 18:00:58.273522 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:58.273477    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d421e35-kq2q2"
Nov 03 18:00:58.672723 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:58.672685    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d421e35-kq2q2"]
Nov 03 18:00:58.676204 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:00:58.676120    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea648b10_b2ec_49ec_8b3d_cc10668990b1.slice/crio-c5bd7b78adb133b00d4aea6b319e0b565ee601a42340407c94e4cb645938d25b WatchSource:0}: Error finding container c5bd7b78adb133b00d4aea6b319e0b565ee601a42340407c94e4cb645938d25b: Status 404 returned error can't find the container with id c5bd7b78adb133b00d4aea6b319e0b565ee601a42340407c94e4cb645938d25b
Nov 03 18:00:59.369974 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:59.369928    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d421e35-kq2q2" event={"ID":"ea648b10-b2ec-49ec-8b3d-cc10668990b1","Type":"ContainerStarted","Data":"1487fe3544bdc6c0b8783724d1b2b0f99a977f4026265935b9684424a5f2f79a"}
Nov 03 18:00:59.370479 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:59.370006    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d421e35-kq2q2" event={"ID":"ea648b10-b2ec-49ec-8b3d-cc10668990b1","Type":"ContainerStarted","Data":"c5bd7b78adb133b00d4aea6b319e0b565ee601a42340407c94e4cb645938d25b"}
Nov 03 18:00:59.481546 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:00:59.481483    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d421e35-kq2q2" podStartSLOduration=2.481469466 podStartE2EDuration="2.481469466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:00:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:00:59.48068686 +0000 UTC m=+898.496884931" watchObservedRunningTime="2026-11-03 18:00:59.481469466 +0000 UTC m=+898.497667535"
Nov 03 18:01:00.378071 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:00.378033    5315 generic.go:356] "Generic (PLEG): container finished" podID="ea648b10-b2ec-49ec-8b3d-cc10668990b1" containerID="1487fe3544bdc6c0b8783724d1b2b0f99a977f4026265935b9684424a5f2f79a" exitCode=0
Nov 03 18:01:00.378522 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:00.378119    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d421e35-kq2q2" event={"ID":"ea648b10-b2ec-49ec-8b3d-cc10668990b1","Type":"ContainerDied","Data":"1487fe3544bdc6c0b8783724d1b2b0f99a977f4026265935b9684424a5f2f79a"}
Nov 03 18:01:01.193128 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:01.193096    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-97p65_c29d200a-c917-46e5-9cf5-21f4d952e470/ovn-acl-logging/0.log"
Nov 03 18:01:01.199520 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:01.199497    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-97p65_c29d200a-c917-46e5-9cf5-21f4d952e470/ovn-acl-logging/0.log"
Nov 03 18:01:01.530643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:01.530593    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d421e35-kq2q2"
Nov 03 18:01:01.596506 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:01.596465    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea648b10-b2ec-49ec-8b3d-cc10668990b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea648b10-b2ec-49ec-8b3d-cc10668990b1-maas-api-xks-ca\") pod \"ea648b10-b2ec-49ec-8b3d-cc10668990b1\" (UID: \"ea648b10-b2ec-49ec-8b3d-cc10668990b1\") "
Nov 03 18:01:01.596711 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:01.596557    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea648b10-b2ec-49ec-8b3d-cc10668990b1-kube-api-access-x47lj\" (UniqueName: \"kubernetes.io/projected/ea648b10-b2ec-49ec-8b3d-cc10668990b1-kube-api-access-x47lj\") pod \"ea648b10-b2ec-49ec-8b3d-cc10668990b1\" (UID: \"ea648b10-b2ec-49ec-8b3d-cc10668990b1\") "
Nov 03 18:01:01.596711 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:01.596591    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea648b10-b2ec-49ec-8b3d-cc10668990b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea648b10-b2ec-49ec-8b3d-cc10668990b1-maas-api-openshift-ca\") pod \"ea648b10-b2ec-49ec-8b3d-cc10668990b1\" (UID: \"ea648b10-b2ec-49ec-8b3d-cc10668990b1\") "
Nov 03 18:01:01.597065 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:01.597033    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea648b10-b2ec-49ec-8b3d-cc10668990b1-maas-api-openshift-ca" pod "ea648b10-b2ec-49ec-8b3d-cc10668990b1" (UID: "ea648b10-b2ec-49ec-8b3d-cc10668990b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:01:01.598713 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:01.598664    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea648b10-b2ec-49ec-8b3d-cc10668990b1-maas-api-xks-ca" pod "ea648b10-b2ec-49ec-8b3d-cc10668990b1" (UID: "ea648b10-b2ec-49ec-8b3d-cc10668990b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:01:01.598871 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:01.598796    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea648b10-b2ec-49ec-8b3d-cc10668990b1-kube-api-access-x47lj" pod "ea648b10-b2ec-49ec-8b3d-cc10668990b1" (UID: "ea648b10-b2ec-49ec-8b3d-cc10668990b1"). InnerVolumeSpecName "kube-api-access-x47lj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:01:01.697754 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:01.697667    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x47lj\" (UniqueName: \"kubernetes.io/projected/ea648b10-b2ec-49ec-8b3d-cc10668990b1-kube-api-access-x47lj\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:01:01.697754 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:01.697701    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea648b10-b2ec-49ec-8b3d-cc10668990b1-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:01:01.697754 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:01.697711    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea648b10-b2ec-49ec-8b3d-cc10668990b1-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:01:02.390513 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:02.390482    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d421e35-kq2q2"
Nov 03 18:01:02.390513 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:02.390507    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d421e35-kq2q2" event={"ID":"ea648b10-b2ec-49ec-8b3d-cc10668990b1","Type":"ContainerDied","Data":"c5bd7b78adb133b00d4aea6b319e0b565ee601a42340407c94e4cb645938d25b"}
Nov 03 18:01:02.390764 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:02.390551    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5bd7b78adb133b00d4aea6b319e0b565ee601a42340407c94e4cb645938d25b"
Nov 03 18:01:06.168694 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:06.168653    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88e3834e-kwk9d"]
Nov 03 18:01:06.169120 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:06.169106    5315 cpu_manager.go:414] "containerMap: removing container" podUID="ea648b10-b2ec-49ec-8b3d-cc10668990b1" containerName="revoke-keys"
Nov 03 18:01:06.169167 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:06.169120    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea648b10-b2ec-49ec-8b3d-cc10668990b1" containerName="revoke-keys"
Nov 03 18:01:06.169205 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:06.169191    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea648b10-b2ec-49ec-8b3d-cc10668990b1" containerName="revoke-keys"
Nov 03 18:01:06.172796 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:06.172771    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e3834e-kwk9d"
Nov 03 18:01:06.226539 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:06.226501    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:01:06.226539 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:06.226540    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:01:06.226770 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:06.226500    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:01:06.246104 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:06.246073    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58247d8d-0abc-4f70-8da3-ff79907ccd82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88e3834e-kwk9d\" (UID: \"58247d8d-0abc-4f70-8da3-ff79907ccd82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e3834e-kwk9d"
Nov 03 18:01:06.246279 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:06.246167    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n528s\" (UniqueName: \"kubernetes.io/projected/58247d8d-0abc-4f70-8da3-ff79907ccd82-kube-api-access-n528s\") pod \"maas-api-revoke-sub-88e3834e-kwk9d\" (UID: \"58247d8d-0abc-4f70-8da3-ff79907ccd82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e3834e-kwk9d"
Nov 03 18:01:06.246351 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:06.246290    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58247d8d-0abc-4f70-8da3-ff79907ccd82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88e3834e-kwk9d\" (UID: \"58247d8d-0abc-4f70-8da3-ff79907ccd82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e3834e-kwk9d"
Nov 03 18:01:06.272525 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:06.272485    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:01:06.274973 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:06.274939    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88e3834e-kwk9d"]
Nov 03 18:01:06.346849 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:06.346817    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n528s\" (UniqueName: \"kubernetes.io/projected/58247d8d-0abc-4f70-8da3-ff79907ccd82-kube-api-access-n528s\") pod \"maas-api-revoke-sub-88e3834e-kwk9d\" (UID: \"58247d8d-0abc-4f70-8da3-ff79907ccd82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e3834e-kwk9d"
Nov 03 18:01:06.347023 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:06.346880    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58247d8d-0abc-4f70-8da3-ff79907ccd82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88e3834e-kwk9d\" (UID: \"58247d8d-0abc-4f70-8da3-ff79907ccd82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e3834e-kwk9d"
Nov 03 18:01:06.347023 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:06.346988    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58247d8d-0abc-4f70-8da3-ff79907ccd82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88e3834e-kwk9d\" (UID: \"58247d8d-0abc-4f70-8da3-ff79907ccd82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e3834e-kwk9d"
Nov 03 18:01:06.347654 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:06.347634    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58247d8d-0abc-4f70-8da3-ff79907ccd82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88e3834e-kwk9d\" (UID: \"58247d8d-0abc-4f70-8da3-ff79907ccd82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e3834e-kwk9d"
Nov 03 18:01:06.349311 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:06.349291    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58247d8d-0abc-4f70-8da3-ff79907ccd82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88e3834e-kwk9d\" (UID: \"58247d8d-0abc-4f70-8da3-ff79907ccd82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e3834e-kwk9d"
Nov 03 18:01:06.404826 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:06.404800    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n528s\" (UniqueName: \"kubernetes.io/projected/58247d8d-0abc-4f70-8da3-ff79907ccd82-kube-api-access-n528s\") pod \"maas-api-revoke-sub-88e3834e-kwk9d\" (UID: \"58247d8d-0abc-4f70-8da3-ff79907ccd82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e3834e-kwk9d"
Nov 03 18:01:06.486689 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:06.486562    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e3834e-kwk9d"
Nov 03 18:01:06.864134 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:06.864099    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88e3834e-kwk9d"]
Nov 03 18:01:06.873152 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:01:06.873122    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58247d8d_0abc_4f70_8da3_ff79907ccd82.slice/crio-4498ab358d0f3cfd26eb4ad71c899ef77330c4741ff3e942cc9ff12c02cd1707 WatchSource:0}: Error finding container 4498ab358d0f3cfd26eb4ad71c899ef77330c4741ff3e942cc9ff12c02cd1707: Status 404 returned error can't find the container with id 4498ab358d0f3cfd26eb4ad71c899ef77330c4741ff3e942cc9ff12c02cd1707
Nov 03 18:01:07.426627 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:07.426519    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e3834e-kwk9d" event={"ID":"58247d8d-0abc-4f70-8da3-ff79907ccd82","Type":"ContainerStarted","Data":"5d1c57937f47cdedfa53466d62f62c47eb29eee45c234c50714044681513988e"}
Nov 03 18:01:07.427065 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:07.426601    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e3834e-kwk9d" event={"ID":"58247d8d-0abc-4f70-8da3-ff79907ccd82","Type":"ContainerStarted","Data":"4498ab358d0f3cfd26eb4ad71c899ef77330c4741ff3e942cc9ff12c02cd1707"}
Nov 03 18:01:07.548104 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:07.548053    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e3834e-kwk9d" podStartSLOduration=1.548040403 podStartE2EDuration="1.548040403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:01:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:01:07.547104121 +0000 UTC m=+906.563302191" watchObservedRunningTime="2026-11-03 18:01:07.548040403 +0000 UTC m=+906.564238489"
Nov 03 18:01:07.781471 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:07.781436    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c53a27e-stzjc"]
Nov 03 18:01:07.785644 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:07.785595    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c53a27e-stzjc"
Nov 03 18:01:07.861288 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:07.861255    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fd12a08-7286-4611-a553-c59ce5e54326-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c53a27e-stzjc\" (UID: \"1fd12a08-7286-4611-a553-c59ce5e54326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c53a27e-stzjc"
Nov 03 18:01:07.861452 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:07.861300    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tvhn\" (UniqueName: \"kubernetes.io/projected/1fd12a08-7286-4611-a553-c59ce5e54326-kube-api-access-9tvhn\") pod \"maas-api-revoke-sub-5c53a27e-stzjc\" (UID: \"1fd12a08-7286-4611-a553-c59ce5e54326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c53a27e-stzjc"
Nov 03 18:01:07.861452 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:07.861325    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fd12a08-7286-4611-a553-c59ce5e54326-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c53a27e-stzjc\" (UID: \"1fd12a08-7286-4611-a553-c59ce5e54326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c53a27e-stzjc"
Nov 03 18:01:07.884397 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:07.884369    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c53a27e-stzjc"]
Nov 03 18:01:07.927585 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:07.927555    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a599aa06-5kd6z"]
Nov 03 18:01:07.931338 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:07.931310    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a599aa06-5kd6z"
Nov 03 18:01:07.962081 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:07.962043    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fd12a08-7286-4611-a553-c59ce5e54326-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c53a27e-stzjc\" (UID: \"1fd12a08-7286-4611-a553-c59ce5e54326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c53a27e-stzjc"
Nov 03 18:01:07.962234 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:07.962096    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tvhn\" (UniqueName: \"kubernetes.io/projected/1fd12a08-7286-4611-a553-c59ce5e54326-kube-api-access-9tvhn\") pod \"maas-api-revoke-sub-5c53a27e-stzjc\" (UID: \"1fd12a08-7286-4611-a553-c59ce5e54326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c53a27e-stzjc"
Nov 03 18:01:07.962234 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:07.962129    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fd12a08-7286-4611-a553-c59ce5e54326-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c53a27e-stzjc\" (UID: \"1fd12a08-7286-4611-a553-c59ce5e54326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c53a27e-stzjc"
Nov 03 18:01:07.962689 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:07.962665    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fd12a08-7286-4611-a553-c59ce5e54326-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c53a27e-stzjc\" (UID: \"1fd12a08-7286-4611-a553-c59ce5e54326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c53a27e-stzjc"
Nov 03 18:01:07.964522 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:07.964496    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fd12a08-7286-4611-a553-c59ce5e54326-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c53a27e-stzjc\" (UID: \"1fd12a08-7286-4611-a553-c59ce5e54326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c53a27e-stzjc"
Nov 03 18:01:08.007567 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:08.007486    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a599aa06-5kd6z"]
Nov 03 18:01:08.063137 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:08.063101    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/091bbf48-d823-4b71-bc36-eb10bc68d8f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a599aa06-5kd6z\" (UID: \"091bbf48-d823-4b71-bc36-eb10bc68d8f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a599aa06-5kd6z"
Nov 03 18:01:08.063340 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:08.063144    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/091bbf48-d823-4b71-bc36-eb10bc68d8f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a599aa06-5kd6z\" (UID: \"091bbf48-d823-4b71-bc36-eb10bc68d8f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a599aa06-5kd6z"
Nov 03 18:01:08.063340 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:08.063310    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smx64\" (UniqueName: \"kubernetes.io/projected/091bbf48-d823-4b71-bc36-eb10bc68d8f7-kube-api-access-smx64\") pod \"maas-api-revoke-sub-a599aa06-5kd6z\" (UID: \"091bbf48-d823-4b71-bc36-eb10bc68d8f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a599aa06-5kd6z"
Nov 03 18:01:08.094225 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:08.094186    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tvhn\" (UniqueName: \"kubernetes.io/projected/1fd12a08-7286-4611-a553-c59ce5e54326-kube-api-access-9tvhn\") pod \"maas-api-revoke-sub-5c53a27e-stzjc\" (UID: \"1fd12a08-7286-4611-a553-c59ce5e54326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c53a27e-stzjc"
Nov 03 18:01:08.117777 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:08.117743    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c53a27e-stzjc"
Nov 03 18:01:08.164954 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:08.164923    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/091bbf48-d823-4b71-bc36-eb10bc68d8f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a599aa06-5kd6z\" (UID: \"091bbf48-d823-4b71-bc36-eb10bc68d8f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a599aa06-5kd6z"
Nov 03 18:01:08.165149 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:08.164969    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/091bbf48-d823-4b71-bc36-eb10bc68d8f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a599aa06-5kd6z\" (UID: \"091bbf48-d823-4b71-bc36-eb10bc68d8f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a599aa06-5kd6z"
Nov 03 18:01:08.165149 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:08.165054    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smx64\" (UniqueName: \"kubernetes.io/projected/091bbf48-d823-4b71-bc36-eb10bc68d8f7-kube-api-access-smx64\") pod \"maas-api-revoke-sub-a599aa06-5kd6z\" (UID: \"091bbf48-d823-4b71-bc36-eb10bc68d8f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a599aa06-5kd6z"
Nov 03 18:01:08.166068 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:08.166002    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/091bbf48-d823-4b71-bc36-eb10bc68d8f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a599aa06-5kd6z\" (UID: \"091bbf48-d823-4b71-bc36-eb10bc68d8f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a599aa06-5kd6z"
Nov 03 18:01:08.168111 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:08.168083    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/091bbf48-d823-4b71-bc36-eb10bc68d8f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a599aa06-5kd6z\" (UID: \"091bbf48-d823-4b71-bc36-eb10bc68d8f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a599aa06-5kd6z"
Nov 03 18:01:08.281697 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:08.281659    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smx64\" (UniqueName: \"kubernetes.io/projected/091bbf48-d823-4b71-bc36-eb10bc68d8f7-kube-api-access-smx64\") pod \"maas-api-revoke-sub-a599aa06-5kd6z\" (UID: \"091bbf48-d823-4b71-bc36-eb10bc68d8f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a599aa06-5kd6z"
Nov 03 18:01:08.434153 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:08.434121    5315 generic.go:356] "Generic (PLEG): container finished" podID="58247d8d-0abc-4f70-8da3-ff79907ccd82" containerID="5d1c57937f47cdedfa53466d62f62c47eb29eee45c234c50714044681513988e" exitCode=0
Nov 03 18:01:08.434524 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:08.434215    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e3834e-kwk9d" event={"ID":"58247d8d-0abc-4f70-8da3-ff79907ccd82","Type":"ContainerDied","Data":"5d1c57937f47cdedfa53466d62f62c47eb29eee45c234c50714044681513988e"}
Nov 03 18:01:08.493882 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:08.493843    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c53a27e-stzjc"]
Nov 03 18:01:08.499234 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:01:08.499198    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fd12a08_7286_4611_a553_c59ce5e54326.slice/crio-6c4a4e01752aee20e928bf48c37fd9d2678853b6cc4f79a967e0044587cf6e47 WatchSource:0}: Error finding container 6c4a4e01752aee20e928bf48c37fd9d2678853b6cc4f79a967e0044587cf6e47: Status 404 returned error can't find the container with id 6c4a4e01752aee20e928bf48c37fd9d2678853b6cc4f79a967e0044587cf6e47
Nov 03 18:01:08.544175 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:08.544149    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a599aa06-5kd6z"
Nov 03 18:01:08.928288 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:08.928250    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a599aa06-5kd6z"]
Nov 03 18:01:08.938525 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:01:08.938493    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod091bbf48_d823_4b71_bc36_eb10bc68d8f7.slice/crio-eedd3e966b5a9bfa01d9e9129597d31bb49adc0739e1e43cd43865f2bb602615 WatchSource:0}: Error finding container eedd3e966b5a9bfa01d9e9129597d31bb49adc0739e1e43cd43865f2bb602615: Status 404 returned error can't find the container with id eedd3e966b5a9bfa01d9e9129597d31bb49adc0739e1e43cd43865f2bb602615
Nov 03 18:01:09.441895 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:09.441864    5315 generic.go:356] "Generic (PLEG): container finished" podID="1fd12a08-7286-4611-a553-c59ce5e54326" containerID="73496c79e43f0cc691a8553671bfcbc8758b3b840c5ffa7666abc35c208f3c4d" exitCode=0
Nov 03 18:01:09.442241 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:09.441946    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c53a27e-stzjc" event={"ID":"1fd12a08-7286-4611-a553-c59ce5e54326","Type":"ContainerDied","Data":"73496c79e43f0cc691a8553671bfcbc8758b3b840c5ffa7666abc35c208f3c4d"}
Nov 03 18:01:09.442241 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:09.442000    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c53a27e-stzjc" event={"ID":"1fd12a08-7286-4611-a553-c59ce5e54326","Type":"ContainerStarted","Data":"6c4a4e01752aee20e928bf48c37fd9d2678853b6cc4f79a967e0044587cf6e47"}
Nov 03 18:01:09.443526 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:09.443499    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a599aa06-5kd6z" event={"ID":"091bbf48-d823-4b71-bc36-eb10bc68d8f7","Type":"ContainerStarted","Data":"449c3097706febf9f9537d4eff7b6e383f8afbf56b014acb0502d2025c0e6f77"}
Nov 03 18:01:09.443662 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:09.443543    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a599aa06-5kd6z" event={"ID":"091bbf48-d823-4b71-bc36-eb10bc68d8f7","Type":"ContainerStarted","Data":"eedd3e966b5a9bfa01d9e9129597d31bb49adc0739e1e43cd43865f2bb602615"}
Nov 03 18:01:09.612170 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:09.612137    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e3834e-kwk9d"
Nov 03 18:01:09.681133 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:09.681030    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a599aa06-5kd6z" podStartSLOduration=2.681017556 podStartE2EDuration="2.681017556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:01:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:01:09.679722902 +0000 UTC m=+908.695920965" watchObservedRunningTime="2026-11-03 18:01:09.681017556 +0000 UTC m=+908.697215673"
Nov 03 18:01:09.803283 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:09.802529    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58247d8d-0abc-4f70-8da3-ff79907ccd82-kube-api-access-n528s\" (UniqueName: \"kubernetes.io/projected/58247d8d-0abc-4f70-8da3-ff79907ccd82-kube-api-access-n528s\") pod \"58247d8d-0abc-4f70-8da3-ff79907ccd82\" (UID: \"58247d8d-0abc-4f70-8da3-ff79907ccd82\") "
Nov 03 18:01:09.803283 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:09.802586    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58247d8d-0abc-4f70-8da3-ff79907ccd82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58247d8d-0abc-4f70-8da3-ff79907ccd82-maas-api-openshift-ca\") pod \"58247d8d-0abc-4f70-8da3-ff79907ccd82\" (UID: \"58247d8d-0abc-4f70-8da3-ff79907ccd82\") "
Nov 03 18:01:09.803283 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:09.802866    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58247d8d-0abc-4f70-8da3-ff79907ccd82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58247d8d-0abc-4f70-8da3-ff79907ccd82-maas-api-xks-ca\") pod \"58247d8d-0abc-4f70-8da3-ff79907ccd82\" (UID: \"58247d8d-0abc-4f70-8da3-ff79907ccd82\") "
Nov 03 18:01:09.808653 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:09.804258    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58247d8d-0abc-4f70-8da3-ff79907ccd82-maas-api-openshift-ca" pod "58247d8d-0abc-4f70-8da3-ff79907ccd82" (UID: "58247d8d-0abc-4f70-8da3-ff79907ccd82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:01:09.822049 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:09.815325    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58247d8d-0abc-4f70-8da3-ff79907ccd82-maas-api-xks-ca" pod "58247d8d-0abc-4f70-8da3-ff79907ccd82" (UID: "58247d8d-0abc-4f70-8da3-ff79907ccd82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:01:09.822049 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:09.815455    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58247d8d-0abc-4f70-8da3-ff79907ccd82-kube-api-access-n528s" pod "58247d8d-0abc-4f70-8da3-ff79907ccd82" (UID: "58247d8d-0abc-4f70-8da3-ff79907ccd82"). InnerVolumeSpecName "kube-api-access-n528s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:01:09.904466 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:09.904417    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n528s\" (UniqueName: \"kubernetes.io/projected/58247d8d-0abc-4f70-8da3-ff79907ccd82-kube-api-access-n528s\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:01:09.904466 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:09.904458    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58247d8d-0abc-4f70-8da3-ff79907ccd82-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:01:09.904731 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:09.904478    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58247d8d-0abc-4f70-8da3-ff79907ccd82-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:01:09.941665 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:09.941529    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx"]
Nov 03 18:01:09.942276 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:09.942245    5315 cpu_manager.go:414] "containerMap: removing container" podUID="58247d8d-0abc-4f70-8da3-ff79907ccd82" containerName="revoke-keys"
Nov 03 18:01:09.942276 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:09.942267    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58247d8d-0abc-4f70-8da3-ff79907ccd82" containerName="revoke-keys"
Nov 03 18:01:09.942436 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:09.942362    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="58247d8d-0abc-4f70-8da3-ff79907ccd82" containerName="revoke-keys"
Nov 03 18:01:09.946004 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:09.945982    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx"
Nov 03 18:01:09.999683 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:09.999648    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-2b4f46-openshift-default-dockercfg-5qdjt\""
Nov 03 18:01:10.015545 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.015514    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx"]
Nov 03 18:01:10.105794 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.105756    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3aa0994-51ec-4076-bc86-c529ad0bb993-istiod-ca-cert\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx\" (UID: \"d3aa0994-51ec-4076-bc86-c529ad0bb993\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx"
Nov 03 18:01:10.105794 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.105799    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3aa0994-51ec-4076-bc86-c529ad0bb993-workload-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx\" (UID: \"d3aa0994-51ec-4076-bc86-c529ad0bb993\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx"
Nov 03 18:01:10.106018 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.105918    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3aa0994-51ec-4076-bc86-c529ad0bb993-istio-podinfo\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx\" (UID: \"d3aa0994-51ec-4076-bc86-c529ad0bb993\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx"
Nov 03 18:01:10.106018 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.105983    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3aa0994-51ec-4076-bc86-c529ad0bb993-istio-token\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx\" (UID: \"d3aa0994-51ec-4076-bc86-c529ad0bb993\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx"
Nov 03 18:01:10.106083 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.106059    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkbsm\" (UniqueName: \"kubernetes.io/projected/d3aa0994-51ec-4076-bc86-c529ad0bb993-kube-api-access-hkbsm\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx\" (UID: \"d3aa0994-51ec-4076-bc86-c529ad0bb993\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx"
Nov 03 18:01:10.106169 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.106085    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3aa0994-51ec-4076-bc86-c529ad0bb993-workload-certs\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx\" (UID: \"d3aa0994-51ec-4076-bc86-c529ad0bb993\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx"
Nov 03 18:01:10.106169 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.106115    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3aa0994-51ec-4076-bc86-c529ad0bb993-istio-envoy\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx\" (UID: \"d3aa0994-51ec-4076-bc86-c529ad0bb993\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx"
Nov 03 18:01:10.106231 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.106199    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3aa0994-51ec-4076-bc86-c529ad0bb993-credential-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx\" (UID: \"d3aa0994-51ec-4076-bc86-c529ad0bb993\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx"
Nov 03 18:01:10.106264 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.106239    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3aa0994-51ec-4076-bc86-c529ad0bb993-istio-data\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx\" (UID: \"d3aa0994-51ec-4076-bc86-c529ad0bb993\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx"
Nov 03 18:01:10.207016 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.206905    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3aa0994-51ec-4076-bc86-c529ad0bb993-istio-data\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx\" (UID: \"d3aa0994-51ec-4076-bc86-c529ad0bb993\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx"
Nov 03 18:01:10.207016 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.206959    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3aa0994-51ec-4076-bc86-c529ad0bb993-istiod-ca-cert\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx\" (UID: \"d3aa0994-51ec-4076-bc86-c529ad0bb993\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx"
Nov 03 18:01:10.207016 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.206989    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3aa0994-51ec-4076-bc86-c529ad0bb993-workload-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx\" (UID: \"d3aa0994-51ec-4076-bc86-c529ad0bb993\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx"
Nov 03 18:01:10.207306 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.207050    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3aa0994-51ec-4076-bc86-c529ad0bb993-istio-podinfo\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx\" (UID: \"d3aa0994-51ec-4076-bc86-c529ad0bb993\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx"
Nov 03 18:01:10.207306 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.207118    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3aa0994-51ec-4076-bc86-c529ad0bb993-istio-token\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx\" (UID: \"d3aa0994-51ec-4076-bc86-c529ad0bb993\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx"
Nov 03 18:01:10.207306 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.207162    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkbsm\" (UniqueName: \"kubernetes.io/projected/d3aa0994-51ec-4076-bc86-c529ad0bb993-kube-api-access-hkbsm\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx\" (UID: \"d3aa0994-51ec-4076-bc86-c529ad0bb993\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx"
Nov 03 18:01:10.207306 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.207187    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3aa0994-51ec-4076-bc86-c529ad0bb993-workload-certs\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx\" (UID: \"d3aa0994-51ec-4076-bc86-c529ad0bb993\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx"
Nov 03 18:01:10.207306 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.207224    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3aa0994-51ec-4076-bc86-c529ad0bb993-istio-envoy\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx\" (UID: \"d3aa0994-51ec-4076-bc86-c529ad0bb993\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx"
Nov 03 18:01:10.207306 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.207284    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3aa0994-51ec-4076-bc86-c529ad0bb993-credential-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx\" (UID: \"d3aa0994-51ec-4076-bc86-c529ad0bb993\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx"
Nov 03 18:01:10.207598 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.207385    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3aa0994-51ec-4076-bc86-c529ad0bb993-istio-data\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx\" (UID: \"d3aa0994-51ec-4076-bc86-c529ad0bb993\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx"
Nov 03 18:01:10.207598 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.207414    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3aa0994-51ec-4076-bc86-c529ad0bb993-workload-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx\" (UID: \"d3aa0994-51ec-4076-bc86-c529ad0bb993\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx"
Nov 03 18:01:10.207749 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.207708    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3aa0994-51ec-4076-bc86-c529ad0bb993-credential-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx\" (UID: \"d3aa0994-51ec-4076-bc86-c529ad0bb993\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx"
Nov 03 18:01:10.207794 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.207710    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3aa0994-51ec-4076-bc86-c529ad0bb993-workload-certs\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx\" (UID: \"d3aa0994-51ec-4076-bc86-c529ad0bb993\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx"
Nov 03 18:01:10.207834 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.207795    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3aa0994-51ec-4076-bc86-c529ad0bb993-istiod-ca-cert\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx\" (UID: \"d3aa0994-51ec-4076-bc86-c529ad0bb993\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx"
Nov 03 18:01:10.209507 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.209486    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3aa0994-51ec-4076-bc86-c529ad0bb993-istio-envoy\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx\" (UID: \"d3aa0994-51ec-4076-bc86-c529ad0bb993\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx"
Nov 03 18:01:10.209825 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.209804    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3aa0994-51ec-4076-bc86-c529ad0bb993-istio-podinfo\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx\" (UID: \"d3aa0994-51ec-4076-bc86-c529ad0bb993\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx"
Nov 03 18:01:10.255932 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.255897    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1048ceff-6bsxw"]
Nov 03 18:01:10.261081 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.261057    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1048ceff-6bsxw"
Nov 03 18:01:10.265856 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.265820    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkbsm\" (UniqueName: \"kubernetes.io/projected/d3aa0994-51ec-4076-bc86-c529ad0bb993-kube-api-access-hkbsm\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx\" (UID: \"d3aa0994-51ec-4076-bc86-c529ad0bb993\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx"
Nov 03 18:01:10.265990 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.265826    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3aa0994-51ec-4076-bc86-c529ad0bb993-istio-token\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx\" (UID: \"d3aa0994-51ec-4076-bc86-c529ad0bb993\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx"
Nov 03 18:01:10.320291 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.320256    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1048ceff-6bsxw"]
Nov 03 18:01:10.409647 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.409593    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f6488e2-977b-4e72-8467-e242a1aaccee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1048ceff-6bsxw\" (UID: \"5f6488e2-977b-4e72-8467-e242a1aaccee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1048ceff-6bsxw"
Nov 03 18:01:10.409814 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.409715    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f6488e2-977b-4e72-8467-e242a1aaccee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1048ceff-6bsxw\" (UID: \"5f6488e2-977b-4e72-8467-e242a1aaccee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1048ceff-6bsxw"
Nov 03 18:01:10.409814 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.409795    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mcnq\" (UniqueName: \"kubernetes.io/projected/5f6488e2-977b-4e72-8467-e242a1aaccee-kube-api-access-2mcnq\") pod \"maas-api-revoke-sub-1048ceff-6bsxw\" (UID: \"5f6488e2-977b-4e72-8467-e242a1aaccee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1048ceff-6bsxw"
Nov 03 18:01:10.451192 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.451156    5315 generic.go:356] "Generic (PLEG): container finished" podID="091bbf48-d823-4b71-bc36-eb10bc68d8f7" containerID="449c3097706febf9f9537d4eff7b6e383f8afbf56b014acb0502d2025c0e6f77" exitCode=0
Nov 03 18:01:10.451588 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.451249    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a599aa06-5kd6z" event={"ID":"091bbf48-d823-4b71-bc36-eb10bc68d8f7","Type":"ContainerDied","Data":"449c3097706febf9f9537d4eff7b6e383f8afbf56b014acb0502d2025c0e6f77"}
Nov 03 18:01:10.452599 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.452575    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e3834e-kwk9d"
Nov 03 18:01:10.452715 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.452631    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e3834e-kwk9d" event={"ID":"58247d8d-0abc-4f70-8da3-ff79907ccd82","Type":"ContainerDied","Data":"4498ab358d0f3cfd26eb4ad71c899ef77330c4741ff3e942cc9ff12c02cd1707"}
Nov 03 18:01:10.452715 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.452676    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4498ab358d0f3cfd26eb4ad71c899ef77330c4741ff3e942cc9ff12c02cd1707"
Nov 03 18:01:10.511436 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.511406    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mcnq\" (UniqueName: \"kubernetes.io/projected/5f6488e2-977b-4e72-8467-e242a1aaccee-kube-api-access-2mcnq\") pod \"maas-api-revoke-sub-1048ceff-6bsxw\" (UID: \"5f6488e2-977b-4e72-8467-e242a1aaccee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1048ceff-6bsxw"
Nov 03 18:01:10.511595 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.511535    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f6488e2-977b-4e72-8467-e242a1aaccee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1048ceff-6bsxw\" (UID: \"5f6488e2-977b-4e72-8467-e242a1aaccee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1048ceff-6bsxw"
Nov 03 18:01:10.511856 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.511603    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f6488e2-977b-4e72-8467-e242a1aaccee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1048ceff-6bsxw\" (UID: \"5f6488e2-977b-4e72-8467-e242a1aaccee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1048ceff-6bsxw"
Nov 03 18:01:10.512932 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.512906    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f6488e2-977b-4e72-8467-e242a1aaccee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1048ceff-6bsxw\" (UID: \"5f6488e2-977b-4e72-8467-e242a1aaccee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1048ceff-6bsxw"
Nov 03 18:01:10.515281 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.515252    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f6488e2-977b-4e72-8467-e242a1aaccee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1048ceff-6bsxw\" (UID: \"5f6488e2-977b-4e72-8467-e242a1aaccee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1048ceff-6bsxw"
Nov 03 18:01:10.558923 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.558891    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx"
Nov 03 18:01:10.573639 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.573598    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mcnq\" (UniqueName: \"kubernetes.io/projected/5f6488e2-977b-4e72-8467-e242a1aaccee-kube-api-access-2mcnq\") pod \"maas-api-revoke-sub-1048ceff-6bsxw\" (UID: \"5f6488e2-977b-4e72-8467-e242a1aaccee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1048ceff-6bsxw"
Nov 03 18:01:10.574327 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.574304    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1048ceff-6bsxw"
Nov 03 18:01:10.606236 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.606207    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c53a27e-stzjc"
Nov 03 18:01:10.714313 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.714222    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fd12a08-7286-4611-a553-c59ce5e54326-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fd12a08-7286-4611-a553-c59ce5e54326-maas-api-xks-ca\") pod \"1fd12a08-7286-4611-a553-c59ce5e54326\" (UID: \"1fd12a08-7286-4611-a553-c59ce5e54326\") "
Nov 03 18:01:10.714313 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.714286    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fd12a08-7286-4611-a553-c59ce5e54326-kube-api-access-9tvhn\" (UniqueName: \"kubernetes.io/projected/1fd12a08-7286-4611-a553-c59ce5e54326-kube-api-access-9tvhn\") pod \"1fd12a08-7286-4611-a553-c59ce5e54326\" (UID: \"1fd12a08-7286-4611-a553-c59ce5e54326\") "
Nov 03 18:01:10.714532 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.714447    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fd12a08-7286-4611-a553-c59ce5e54326-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fd12a08-7286-4611-a553-c59ce5e54326-maas-api-openshift-ca\") pod \"1fd12a08-7286-4611-a553-c59ce5e54326\" (UID: \"1fd12a08-7286-4611-a553-c59ce5e54326\") "
Nov 03 18:01:10.714983 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.714955    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fd12a08-7286-4611-a553-c59ce5e54326-maas-api-openshift-ca" pod "1fd12a08-7286-4611-a553-c59ce5e54326" (UID: "1fd12a08-7286-4611-a553-c59ce5e54326"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:01:10.716870 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.716842    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fd12a08-7286-4611-a553-c59ce5e54326-maas-api-xks-ca" pod "1fd12a08-7286-4611-a553-c59ce5e54326" (UID: "1fd12a08-7286-4611-a553-c59ce5e54326"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:01:10.717277 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.717254    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fd12a08-7286-4611-a553-c59ce5e54326-kube-api-access-9tvhn" pod "1fd12a08-7286-4611-a553-c59ce5e54326" (UID: "1fd12a08-7286-4611-a553-c59ce5e54326"). InnerVolumeSpecName "kube-api-access-9tvhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:01:10.833799 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.832074    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fd12a08-7286-4611-a553-c59ce5e54326-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:01:10.833799 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.832108    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fd12a08-7286-4611-a553-c59ce5e54326-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:01:10.833799 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:10.832124    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tvhn\" (UniqueName: \"kubernetes.io/projected/1fd12a08-7286-4611-a553-c59ce5e54326-kube-api-access-9tvhn\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:01:11.010123 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:11.010029    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx"]
Nov 03 18:01:11.012987 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:11.012965    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1048ceff-6bsxw"]
Nov 03 18:01:11.016316 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:01:11.016283    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3aa0994_51ec_4076_bc86_c529ad0bb993.slice/crio-84189e3f98f05c87b0d924fbb6e61df08aba83a275fe58c4962ba38ea9f8b61b WatchSource:0}: Error finding container 84189e3f98f05c87b0d924fbb6e61df08aba83a275fe58c4962ba38ea9f8b61b: Status 404 returned error can't find the container with id 84189e3f98f05c87b0d924fbb6e61df08aba83a275fe58c4962ba38ea9f8b61b
Nov 03 18:01:11.019305 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:11.019271    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:01:11.019395 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:11.019362    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:01:11.019448 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:11.019397    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:01:11.019590 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:01:11.019566    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f6488e2_977b_4e72_8467_e242a1aaccee.slice/crio-99e9c91e14a95c7f9efb84ef5b74fe51feaaeaee92e0dd101de0ae915a8b883b WatchSource:0}: Error finding container 99e9c91e14a95c7f9efb84ef5b74fe51feaaeaee92e0dd101de0ae915a8b883b: Status 404 returned error can't find the container with id 99e9c91e14a95c7f9efb84ef5b74fe51feaaeaee92e0dd101de0ae915a8b883b
Nov 03 18:01:11.459416 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:11.459378    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx" event={"ID":"d3aa0994-51ec-4076-bc86-c529ad0bb993","Type":"ContainerStarted","Data":"f483f014e8dc90ff635ed6820c3298a9734aefa3720a84ca1339fe199273932d"}
Nov 03 18:01:11.459889 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:11.459442    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx" event={"ID":"d3aa0994-51ec-4076-bc86-c529ad0bb993","Type":"ContainerStarted","Data":"84189e3f98f05c87b0d924fbb6e61df08aba83a275fe58c4962ba38ea9f8b61b"}
Nov 03 18:01:11.460873 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:11.460853    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c53a27e-stzjc"
Nov 03 18:01:11.460873 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:11.460863    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c53a27e-stzjc" event={"ID":"1fd12a08-7286-4611-a553-c59ce5e54326","Type":"ContainerDied","Data":"6c4a4e01752aee20e928bf48c37fd9d2678853b6cc4f79a967e0044587cf6e47"}
Nov 03 18:01:11.461000 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:11.460903    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c4a4e01752aee20e928bf48c37fd9d2678853b6cc4f79a967e0044587cf6e47"
Nov 03 18:01:11.462590 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:11.462560    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1048ceff-6bsxw" event={"ID":"5f6488e2-977b-4e72-8467-e242a1aaccee","Type":"ContainerStarted","Data":"9db42ad22a2ceb767968f1d11b04743dfd2459a596ad9347b29b3183f51b3c5b"}
Nov 03 18:01:11.462739 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:11.462630    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1048ceff-6bsxw" event={"ID":"5f6488e2-977b-4e72-8467-e242a1aaccee","Type":"ContainerStarted","Data":"99e9c91e14a95c7f9efb84ef5b74fe51feaaeaee92e0dd101de0ae915a8b883b"}
Nov 03 18:01:11.559742 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:11.559699    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx"
Nov 03 18:01:11.584642 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:11.584542    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx" podStartSLOduration=2.584530131 podStartE2EDuration="2.584530131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:01:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:01:11.583547964 +0000 UTC m=+910.599746033" watchObservedRunningTime="2026-11-03 18:01:11.584530131 +0000 UTC m=+910.600728200"
Nov 03 18:01:11.606037 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:11.606011    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a599aa06-5kd6z"
Nov 03 18:01:11.640366 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:11.640336    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/091bbf48-d823-4b71-bc36-eb10bc68d8f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/091bbf48-d823-4b71-bc36-eb10bc68d8f7-maas-api-openshift-ca\") pod \"091bbf48-d823-4b71-bc36-eb10bc68d8f7\" (UID: \"091bbf48-d823-4b71-bc36-eb10bc68d8f7\") "
Nov 03 18:01:11.640529 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:11.640374    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/091bbf48-d823-4b71-bc36-eb10bc68d8f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/091bbf48-d823-4b71-bc36-eb10bc68d8f7-maas-api-xks-ca\") pod \"091bbf48-d823-4b71-bc36-eb10bc68d8f7\" (UID: \"091bbf48-d823-4b71-bc36-eb10bc68d8f7\") "
Nov 03 18:01:11.640529 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:11.640495    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/091bbf48-d823-4b71-bc36-eb10bc68d8f7-kube-api-access-smx64\" (UniqueName: \"kubernetes.io/projected/091bbf48-d823-4b71-bc36-eb10bc68d8f7-kube-api-access-smx64\") pod \"091bbf48-d823-4b71-bc36-eb10bc68d8f7\" (UID: \"091bbf48-d823-4b71-bc36-eb10bc68d8f7\") "
Nov 03 18:01:11.640731 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:11.640710    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/091bbf48-d823-4b71-bc36-eb10bc68d8f7-maas-api-openshift-ca" pod "091bbf48-d823-4b71-bc36-eb10bc68d8f7" (UID: "091bbf48-d823-4b71-bc36-eb10bc68d8f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:01:11.640897 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:11.640883    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/091bbf48-d823-4b71-bc36-eb10bc68d8f7-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:01:11.642500 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:11.642477    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/091bbf48-d823-4b71-bc36-eb10bc68d8f7-kube-api-access-smx64" pod "091bbf48-d823-4b71-bc36-eb10bc68d8f7" (UID: "091bbf48-d823-4b71-bc36-eb10bc68d8f7"). InnerVolumeSpecName "kube-api-access-smx64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:01:11.642634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:11.642545    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/091bbf48-d823-4b71-bc36-eb10bc68d8f7-maas-api-xks-ca" pod "091bbf48-d823-4b71-bc36-eb10bc68d8f7" (UID: "091bbf48-d823-4b71-bc36-eb10bc68d8f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:01:11.694329 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:11.694282    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1048ceff-6bsxw" podStartSLOduration=1.694268528 podStartE2EDuration="1.694268528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:01:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:01:11.692730872 +0000 UTC m=+910.708928942" watchObservedRunningTime="2026-11-03 18:01:11.694268528 +0000 UTC m=+910.710466597"
Nov 03 18:01:11.741459 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:11.741423    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/091bbf48-d823-4b71-bc36-eb10bc68d8f7-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:01:11.741459 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:11.741455    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smx64\" (UniqueName: \"kubernetes.io/projected/091bbf48-d823-4b71-bc36-eb10bc68d8f7-kube-api-access-smx64\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:01:12.469749 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:12.469712    5315 generic.go:356] "Generic (PLEG): container finished" podID="5f6488e2-977b-4e72-8467-e242a1aaccee" containerID="9db42ad22a2ceb767968f1d11b04743dfd2459a596ad9347b29b3183f51b3c5b" exitCode=0
Nov 03 18:01:12.470231 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:12.469793    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1048ceff-6bsxw" event={"ID":"5f6488e2-977b-4e72-8467-e242a1aaccee","Type":"ContainerDied","Data":"9db42ad22a2ceb767968f1d11b04743dfd2459a596ad9347b29b3183f51b3c5b"}
Nov 03 18:01:12.471277 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:12.471256    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a599aa06-5kd6z"
Nov 03 18:01:12.471384 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:12.471291    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a599aa06-5kd6z" event={"ID":"091bbf48-d823-4b71-bc36-eb10bc68d8f7","Type":"ContainerDied","Data":"eedd3e966b5a9bfa01d9e9129597d31bb49adc0739e1e43cd43865f2bb602615"}
Nov 03 18:01:12.471384 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:12.471330    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eedd3e966b5a9bfa01d9e9129597d31bb49adc0739e1e43cd43865f2bb602615"
Nov 03 18:01:12.561547 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:12.561516    5315 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 18:01:12.561713 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:12.561579    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx" podUID="d3aa0994-51ec-4076-bc86-c529ad0bb993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 18:01:13.577571 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:13.577536    5315 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 18:01:13.578162 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:13.578131    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx" podUID="d3aa0994-51ec-4076-bc86-c529ad0bb993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 18:01:13.810728 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:13.810693    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1048ceff-6bsxw"
Nov 03 18:01:13.902635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:13.896966    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f6488e2-977b-4e72-8467-e242a1aaccee-kube-api-access-2mcnq\" (UniqueName: \"kubernetes.io/projected/5f6488e2-977b-4e72-8467-e242a1aaccee-kube-api-access-2mcnq\") pod \"5f6488e2-977b-4e72-8467-e242a1aaccee\" (UID: \"5f6488e2-977b-4e72-8467-e242a1aaccee\") "
Nov 03 18:01:13.902635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:13.897081    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f6488e2-977b-4e72-8467-e242a1aaccee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f6488e2-977b-4e72-8467-e242a1aaccee-maas-api-openshift-ca\") pod \"5f6488e2-977b-4e72-8467-e242a1aaccee\" (UID: \"5f6488e2-977b-4e72-8467-e242a1aaccee\") "
Nov 03 18:01:13.902635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:13.897172    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f6488e2-977b-4e72-8467-e242a1aaccee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f6488e2-977b-4e72-8467-e242a1aaccee-maas-api-xks-ca\") pod \"5f6488e2-977b-4e72-8467-e242a1aaccee\" (UID: \"5f6488e2-977b-4e72-8467-e242a1aaccee\") "
Nov 03 18:01:13.902635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:13.898398    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f6488e2-977b-4e72-8467-e242a1aaccee-maas-api-openshift-ca" pod "5f6488e2-977b-4e72-8467-e242a1aaccee" (UID: "5f6488e2-977b-4e72-8467-e242a1aaccee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:01:13.902635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:13.901925    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f6488e2-977b-4e72-8467-e242a1aaccee-maas-api-xks-ca" pod "5f6488e2-977b-4e72-8467-e242a1aaccee" (UID: "5f6488e2-977b-4e72-8467-e242a1aaccee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:01:13.902635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:13.902103    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f6488e2-977b-4e72-8467-e242a1aaccee-kube-api-access-2mcnq" pod "5f6488e2-977b-4e72-8467-e242a1aaccee" (UID: "5f6488e2-977b-4e72-8467-e242a1aaccee"). InnerVolumeSpecName "kube-api-access-2mcnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:01:14.009691 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:14.009350    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f6488e2-977b-4e72-8467-e242a1aaccee-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:01:14.009691 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:14.009393    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f6488e2-977b-4e72-8467-e242a1aaccee-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:01:14.009691 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:14.009411    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mcnq\" (UniqueName: \"kubernetes.io/projected/5f6488e2-977b-4e72-8467-e242a1aaccee-kube-api-access-2mcnq\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:01:14.488355 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:14.488319    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1048ceff-6bsxw" event={"ID":"5f6488e2-977b-4e72-8467-e242a1aaccee","Type":"ContainerDied","Data":"99e9c91e14a95c7f9efb84ef5b74fe51feaaeaee92e0dd101de0ae915a8b883b"}
Nov 03 18:01:14.488355 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:14.488337    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1048ceff-6bsxw"
Nov 03 18:01:14.488538 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:14.488369    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99e9c91e14a95c7f9efb84ef5b74fe51feaaeaee92e0dd101de0ae915a8b883b"
Nov 03 18:01:14.560225 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:14.560190    5315 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 18:01:14.560381 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:14.560262    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx" podUID="d3aa0994-51ec-4076-bc86-c529ad0bb993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 18:01:15.560030 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:15.559985    5315 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 18:01:15.560439 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:15.560059    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx" podUID="d3aa0994-51ec-4076-bc86-c529ad0bb993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 18:01:15.785470 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:15.785423    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d0be55c-rzk6z"]
Nov 03 18:01:15.785902 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:15.785888    5315 cpu_manager.go:414] "containerMap: removing container" podUID="091bbf48-d823-4b71-bc36-eb10bc68d8f7" containerName="revoke-keys"
Nov 03 18:01:15.785967 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:15.785903    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="091bbf48-d823-4b71-bc36-eb10bc68d8f7" containerName="revoke-keys"
Nov 03 18:01:15.785967 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:15.785926    5315 cpu_manager.go:414] "containerMap: removing container" podUID="5f6488e2-977b-4e72-8467-e242a1aaccee" containerName="revoke-keys"
Nov 03 18:01:15.785967 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:15.785931    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f6488e2-977b-4e72-8467-e242a1aaccee" containerName="revoke-keys"
Nov 03 18:01:15.785967 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:15.785944    5315 cpu_manager.go:414] "containerMap: removing container" podUID="1fd12a08-7286-4611-a553-c59ce5e54326" containerName="revoke-keys"
Nov 03 18:01:15.785967 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:15.785949    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fd12a08-7286-4611-a553-c59ce5e54326" containerName="revoke-keys"
Nov 03 18:01:15.786133 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:15.786028    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f6488e2-977b-4e72-8467-e242a1aaccee" containerName="revoke-keys"
Nov 03 18:01:15.786133 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:15.786039    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fd12a08-7286-4611-a553-c59ce5e54326" containerName="revoke-keys"
Nov 03 18:01:15.786133 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:15.786049    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="091bbf48-d823-4b71-bc36-eb10bc68d8f7" containerName="revoke-keys"
Nov 03 18:01:15.789779 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:15.789753    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d0be55c-rzk6z"
Nov 03 18:01:15.849183 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:15.838958    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:01:15.849183 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:15.839115    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:01:15.868400 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:15.868359    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a12e0e65-73d8-4441-873f-df3978fc0a04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d0be55c-rzk6z\" (UID: \"a12e0e65-73d8-4441-873f-df3978fc0a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d0be55c-rzk6z"
Nov 03 18:01:15.868400 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:15.868402    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkv9z\" (UniqueName: \"kubernetes.io/projected/a12e0e65-73d8-4441-873f-df3978fc0a04-kube-api-access-xkv9z\") pod \"maas-api-revoke-sub-9d0be55c-rzk6z\" (UID: \"a12e0e65-73d8-4441-873f-df3978fc0a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d0be55c-rzk6z"
Nov 03 18:01:15.868704 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:15.868488    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a12e0e65-73d8-4441-873f-df3978fc0a04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d0be55c-rzk6z\" (UID: \"a12e0e65-73d8-4441-873f-df3978fc0a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d0be55c-rzk6z"
Nov 03 18:01:15.885112 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:15.885080    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:01:15.885112 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:15.885083    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:01:15.887726 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:15.887700    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d0be55c-rzk6z"]
Nov 03 18:01:15.969240 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:15.969202    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a12e0e65-73d8-4441-873f-df3978fc0a04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d0be55c-rzk6z\" (UID: \"a12e0e65-73d8-4441-873f-df3978fc0a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d0be55c-rzk6z"
Nov 03 18:01:15.969240 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:15.969245    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkv9z\" (UniqueName: \"kubernetes.io/projected/a12e0e65-73d8-4441-873f-df3978fc0a04-kube-api-access-xkv9z\") pod \"maas-api-revoke-sub-9d0be55c-rzk6z\" (UID: \"a12e0e65-73d8-4441-873f-df3978fc0a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d0be55c-rzk6z"
Nov 03 18:01:15.969462 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:15.969275    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a12e0e65-73d8-4441-873f-df3978fc0a04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d0be55c-rzk6z\" (UID: \"a12e0e65-73d8-4441-873f-df3978fc0a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d0be55c-rzk6z"
Nov 03 18:01:15.969929 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:15.969904    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a12e0e65-73d8-4441-873f-df3978fc0a04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d0be55c-rzk6z\" (UID: \"a12e0e65-73d8-4441-873f-df3978fc0a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d0be55c-rzk6z"
Nov 03 18:01:15.971634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:15.971602    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a12e0e65-73d8-4441-873f-df3978fc0a04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d0be55c-rzk6z\" (UID: \"a12e0e65-73d8-4441-873f-df3978fc0a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d0be55c-rzk6z"
Nov 03 18:01:16.048629 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:16.044215    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkv9z\" (UniqueName: \"kubernetes.io/projected/a12e0e65-73d8-4441-873f-df3978fc0a04-kube-api-access-xkv9z\") pod \"maas-api-revoke-sub-9d0be55c-rzk6z\" (UID: \"a12e0e65-73d8-4441-873f-df3978fc0a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d0be55c-rzk6z"
Nov 03 18:01:16.103862 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:16.103783    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d0be55c-rzk6z"
Nov 03 18:01:16.506441 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:16.506341    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d0be55c-rzk6z"]
Nov 03 18:01:16.512232 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:01:16.512194    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda12e0e65_73d8_4441_873f_df3978fc0a04.slice/crio-3d2c0543b1dfedf8012ab451cca16870e69d96e105f7e14a824004fdcec9a442 WatchSource:0}: Error finding container 3d2c0543b1dfedf8012ab451cca16870e69d96e105f7e14a824004fdcec9a442: Status 404 returned error can't find the container with id 3d2c0543b1dfedf8012ab451cca16870e69d96e105f7e14a824004fdcec9a442
Nov 03 18:01:16.559319 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:16.559290    5315 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 18:01:16.559442 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:16.559342    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx" podUID="d3aa0994-51ec-4076-bc86-c529ad0bb993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 18:01:17.508909 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:17.508872    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d0be55c-rzk6z" event={"ID":"a12e0e65-73d8-4441-873f-df3978fc0a04","Type":"ContainerStarted","Data":"641315f6c7fd9f903a02c40ac1a7d47abba2e98d7db410274656840b234e909f"}
Nov 03 18:01:17.509569 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:17.508929    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d0be55c-rzk6z" event={"ID":"a12e0e65-73d8-4441-873f-df3978fc0a04","Type":"ContainerStarted","Data":"3d2c0543b1dfedf8012ab451cca16870e69d96e105f7e14a824004fdcec9a442"}
Nov 03 18:01:17.559641 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:17.559586    5315 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 18:01:17.559810 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:17.559673    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx" podUID="d3aa0994-51ec-4076-bc86-c529ad0bb993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 18:01:17.620948 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:17.620888    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d0be55c-rzk6z" podStartSLOduration=2.620871751 podStartE2EDuration="2.620871751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:01:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:01:17.618979499 +0000 UTC m=+916.635177570" watchObservedRunningTime="2026-11-03 18:01:17.620871751 +0000 UTC m=+916.637069860"
Nov 03 18:01:18.516654 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:18.516593    5315 generic.go:356] "Generic (PLEG): container finished" podID="a12e0e65-73d8-4441-873f-df3978fc0a04" containerID="641315f6c7fd9f903a02c40ac1a7d47abba2e98d7db410274656840b234e909f" exitCode=0
Nov 03 18:01:18.517025 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:18.516684    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d0be55c-rzk6z" event={"ID":"a12e0e65-73d8-4441-873f-df3978fc0a04","Type":"ContainerDied","Data":"641315f6c7fd9f903a02c40ac1a7d47abba2e98d7db410274656840b234e909f"}
Nov 03 18:01:18.675250 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:18.675215    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f88012a-7nkk7"]
Nov 03 18:01:18.678917 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:18.678891    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f88012a-7nkk7"
Nov 03 18:01:18.706325 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:18.706295    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94c7f5f6-0b51-48a0-b7b6-81edc7bff526-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f88012a-7nkk7\" (UID: \"94c7f5f6-0b51-48a0-b7b6-81edc7bff526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f88012a-7nkk7"
Nov 03 18:01:18.706468 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:18.706446    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlqpj\" (UniqueName: \"kubernetes.io/projected/94c7f5f6-0b51-48a0-b7b6-81edc7bff526-kube-api-access-dlqpj\") pod \"maas-api-revoke-sub-8f88012a-7nkk7\" (UID: \"94c7f5f6-0b51-48a0-b7b6-81edc7bff526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f88012a-7nkk7"
Nov 03 18:01:18.706521 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:18.706506    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94c7f5f6-0b51-48a0-b7b6-81edc7bff526-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f88012a-7nkk7\" (UID: \"94c7f5f6-0b51-48a0-b7b6-81edc7bff526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f88012a-7nkk7"
Nov 03 18:01:18.749866 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:18.749821    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f88012a-7nkk7"]
Nov 03 18:01:18.807714 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:18.807678    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94c7f5f6-0b51-48a0-b7b6-81edc7bff526-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f88012a-7nkk7\" (UID: \"94c7f5f6-0b51-48a0-b7b6-81edc7bff526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f88012a-7nkk7"
Nov 03 18:01:18.807929 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:18.807779    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94c7f5f6-0b51-48a0-b7b6-81edc7bff526-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f88012a-7nkk7\" (UID: \"94c7f5f6-0b51-48a0-b7b6-81edc7bff526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f88012a-7nkk7"
Nov 03 18:01:18.807929 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:18.807912    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlqpj\" (UniqueName: \"kubernetes.io/projected/94c7f5f6-0b51-48a0-b7b6-81edc7bff526-kube-api-access-dlqpj\") pod \"maas-api-revoke-sub-8f88012a-7nkk7\" (UID: \"94c7f5f6-0b51-48a0-b7b6-81edc7bff526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f88012a-7nkk7"
Nov 03 18:01:18.808342 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:18.808316    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94c7f5f6-0b51-48a0-b7b6-81edc7bff526-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f88012a-7nkk7\" (UID: \"94c7f5f6-0b51-48a0-b7b6-81edc7bff526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f88012a-7nkk7"
Nov 03 18:01:18.810134 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:18.810111    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94c7f5f6-0b51-48a0-b7b6-81edc7bff526-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f88012a-7nkk7\" (UID: \"94c7f5f6-0b51-48a0-b7b6-81edc7bff526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f88012a-7nkk7"
Nov 03 18:01:18.865815 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:18.865780    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlqpj\" (UniqueName: \"kubernetes.io/projected/94c7f5f6-0b51-48a0-b7b6-81edc7bff526-kube-api-access-dlqpj\") pod \"maas-api-revoke-sub-8f88012a-7nkk7\" (UID: \"94c7f5f6-0b51-48a0-b7b6-81edc7bff526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f88012a-7nkk7"
Nov 03 18:01:18.870197 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:18.870169    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx"
Nov 03 18:01:18.870546 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:18.870524    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx"
Nov 03 18:01:18.871206 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:18.871187    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx"
Nov 03 18:01:18.996290 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:18.996257    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f88012a-7nkk7"
Nov 03 18:01:19.369598 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:19.369568    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f88012a-7nkk7"]
Nov 03 18:01:19.372409 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:01:19.372367    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94c7f5f6_0b51_48a0_b7b6_81edc7bff526.slice/crio-d43be26cccc7823b8befa125bf6ec5a838bf91d2311a0461aa1cb3a2fde03b1b WatchSource:0}: Error finding container d43be26cccc7823b8befa125bf6ec5a838bf91d2311a0461aa1cb3a2fde03b1b: Status 404 returned error can't find the container with id d43be26cccc7823b8befa125bf6ec5a838bf91d2311a0461aa1cb3a2fde03b1b
Nov 03 18:01:19.523244 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:19.523206    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f88012a-7nkk7" event={"ID":"94c7f5f6-0b51-48a0-b7b6-81edc7bff526","Type":"ContainerStarted","Data":"d43be26cccc7823b8befa125bf6ec5a838bf91d2311a0461aa1cb3a2fde03b1b"}
Nov 03 18:01:19.643769 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:19.643744    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d0be55c-rzk6z"
Nov 03 18:01:19.718933 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:19.718900    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a12e0e65-73d8-4441-873f-df3978fc0a04-kube-api-access-xkv9z\" (UniqueName: \"kubernetes.io/projected/a12e0e65-73d8-4441-873f-df3978fc0a04-kube-api-access-xkv9z\") pod \"a12e0e65-73d8-4441-873f-df3978fc0a04\" (UID: \"a12e0e65-73d8-4441-873f-df3978fc0a04\") "
Nov 03 18:01:19.719102 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:19.718974    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a12e0e65-73d8-4441-873f-df3978fc0a04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a12e0e65-73d8-4441-873f-df3978fc0a04-maas-api-openshift-ca\") pod \"a12e0e65-73d8-4441-873f-df3978fc0a04\" (UID: \"a12e0e65-73d8-4441-873f-df3978fc0a04\") "
Nov 03 18:01:19.719102 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:19.719076    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a12e0e65-73d8-4441-873f-df3978fc0a04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a12e0e65-73d8-4441-873f-df3978fc0a04-maas-api-xks-ca\") pod \"a12e0e65-73d8-4441-873f-df3978fc0a04\" (UID: \"a12e0e65-73d8-4441-873f-df3978fc0a04\") "
Nov 03 18:01:19.719347 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:19.719326    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a12e0e65-73d8-4441-873f-df3978fc0a04-maas-api-openshift-ca" pod "a12e0e65-73d8-4441-873f-df3978fc0a04" (UID: "a12e0e65-73d8-4441-873f-df3978fc0a04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:01:19.719569 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:19.719546    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a12e0e65-73d8-4441-873f-df3978fc0a04-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:01:19.721477 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:19.721448    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a12e0e65-73d8-4441-873f-df3978fc0a04-maas-api-xks-ca" pod "a12e0e65-73d8-4441-873f-df3978fc0a04" (UID: "a12e0e65-73d8-4441-873f-df3978fc0a04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:01:19.721477 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:19.721464    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a12e0e65-73d8-4441-873f-df3978fc0a04-kube-api-access-xkv9z" pod "a12e0e65-73d8-4441-873f-df3978fc0a04" (UID: "a12e0e65-73d8-4441-873f-df3978fc0a04"). InnerVolumeSpecName "kube-api-access-xkv9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:01:19.820421 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:19.820392    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkv9z\" (UniqueName: \"kubernetes.io/projected/a12e0e65-73d8-4441-873f-df3978fc0a04-kube-api-access-xkv9z\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:01:19.820421 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:19.820422    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a12e0e65-73d8-4441-873f-df3978fc0a04-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:01:20.530720 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:20.530684    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d0be55c-rzk6z"
Nov 03 18:01:20.530720 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:20.530708    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d0be55c-rzk6z" event={"ID":"a12e0e65-73d8-4441-873f-df3978fc0a04","Type":"ContainerDied","Data":"3d2c0543b1dfedf8012ab451cca16870e69d96e105f7e14a824004fdcec9a442"}
Nov 03 18:01:20.531247 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:20.530763    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d2c0543b1dfedf8012ab451cca16870e69d96e105f7e14a824004fdcec9a442"
Nov 03 18:01:20.532545 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:20.532516    5315 generic.go:356] "Generic (PLEG): container finished" podID="94c7f5f6-0b51-48a0-b7b6-81edc7bff526" containerID="18fe9b6516b7171208776b04259ec7b7f3aacafc6410d9e9849d6ea1b6df1d88" exitCode=0
Nov 03 18:01:20.532746 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:20.532624    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f88012a-7nkk7" event={"ID":"94c7f5f6-0b51-48a0-b7b6-81edc7bff526","Type":"ContainerDied","Data":"18fe9b6516b7171208776b04259ec7b7f3aacafc6410d9e9849d6ea1b6df1d88"}
Nov 03 18:01:21.026131 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:21.026105    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e91a8237-vvgc5"]
Nov 03 18:01:21.026590 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:21.026576    5315 cpu_manager.go:414] "containerMap: removing container" podUID="a12e0e65-73d8-4441-873f-df3978fc0a04" containerName="revoke-keys"
Nov 03 18:01:21.026648 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:21.026591    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a12e0e65-73d8-4441-873f-df3978fc0a04" containerName="revoke-keys"
Nov 03 18:01:21.026689 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:21.026679    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="a12e0e65-73d8-4441-873f-df3978fc0a04" containerName="revoke-keys"
Nov 03 18:01:21.030234 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:21.030206    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e91a8237-vvgc5"
Nov 03 18:01:21.104466 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:21.104431    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e91a8237-vvgc5"]
Nov 03 18:01:21.132906 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:21.132860    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8qkt\" (UniqueName: \"kubernetes.io/projected/f2943536-963c-462e-9715-7756e5af31f6-kube-api-access-x8qkt\") pod \"maas-api-revoke-sub-e91a8237-vvgc5\" (UID: \"f2943536-963c-462e-9715-7756e5af31f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e91a8237-vvgc5"
Nov 03 18:01:21.133105 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:21.132914    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2943536-963c-462e-9715-7756e5af31f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e91a8237-vvgc5\" (UID: \"f2943536-963c-462e-9715-7756e5af31f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e91a8237-vvgc5"
Nov 03 18:01:21.133182 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:21.133123    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2943536-963c-462e-9715-7756e5af31f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e91a8237-vvgc5\" (UID: \"f2943536-963c-462e-9715-7756e5af31f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e91a8237-vvgc5"
Nov 03 18:01:21.234554 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:21.234508    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8qkt\" (UniqueName: \"kubernetes.io/projected/f2943536-963c-462e-9715-7756e5af31f6-kube-api-access-x8qkt\") pod \"maas-api-revoke-sub-e91a8237-vvgc5\" (UID: \"f2943536-963c-462e-9715-7756e5af31f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e91a8237-vvgc5"
Nov 03 18:01:21.234744 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:21.234562    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2943536-963c-462e-9715-7756e5af31f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e91a8237-vvgc5\" (UID: \"f2943536-963c-462e-9715-7756e5af31f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e91a8237-vvgc5"
Nov 03 18:01:21.234744 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:21.234630    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2943536-963c-462e-9715-7756e5af31f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e91a8237-vvgc5\" (UID: \"f2943536-963c-462e-9715-7756e5af31f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e91a8237-vvgc5"
Nov 03 18:01:21.235216 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:21.235184    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2943536-963c-462e-9715-7756e5af31f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e91a8237-vvgc5\" (UID: \"f2943536-963c-462e-9715-7756e5af31f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e91a8237-vvgc5"
Nov 03 18:01:21.236906 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:21.236881    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2943536-963c-462e-9715-7756e5af31f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e91a8237-vvgc5\" (UID: \"f2943536-963c-462e-9715-7756e5af31f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e91a8237-vvgc5"
Nov 03 18:01:21.293030 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:21.292956    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8qkt\" (UniqueName: \"kubernetes.io/projected/f2943536-963c-462e-9715-7756e5af31f6-kube-api-access-x8qkt\") pod \"maas-api-revoke-sub-e91a8237-vvgc5\" (UID: \"f2943536-963c-462e-9715-7756e5af31f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e91a8237-vvgc5"
Nov 03 18:01:21.343344 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:21.343312    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e91a8237-vvgc5"
Nov 03 18:01:21.684338 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:21.684310    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f88012a-7nkk7"
Nov 03 18:01:21.718849 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:21.718816    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e91a8237-vvgc5"]
Nov 03 18:01:21.724864 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:01:21.724837    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2943536_963c_462e_9715_7756e5af31f6.slice/crio-a212ec5eb7251288504295ea27b79f26e66125117e4065a0cc2819e016d519cc WatchSource:0}: Error finding container a212ec5eb7251288504295ea27b79f26e66125117e4065a0cc2819e016d519cc: Status 404 returned error can't find the container with id a212ec5eb7251288504295ea27b79f26e66125117e4065a0cc2819e016d519cc
Nov 03 18:01:21.739993 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:21.739967    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94c7f5f6-0b51-48a0-b7b6-81edc7bff526-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94c7f5f6-0b51-48a0-b7b6-81edc7bff526-maas-api-openshift-ca\") pod \"94c7f5f6-0b51-48a0-b7b6-81edc7bff526\" (UID: \"94c7f5f6-0b51-48a0-b7b6-81edc7bff526\") "
Nov 03 18:01:21.740098 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:21.740057    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94c7f5f6-0b51-48a0-b7b6-81edc7bff526-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94c7f5f6-0b51-48a0-b7b6-81edc7bff526-maas-api-xks-ca\") pod \"94c7f5f6-0b51-48a0-b7b6-81edc7bff526\" (UID: \"94c7f5f6-0b51-48a0-b7b6-81edc7bff526\") "
Nov 03 18:01:21.740157 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:21.740121    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94c7f5f6-0b51-48a0-b7b6-81edc7bff526-kube-api-access-dlqpj\" (UniqueName: \"kubernetes.io/projected/94c7f5f6-0b51-48a0-b7b6-81edc7bff526-kube-api-access-dlqpj\") pod \"94c7f5f6-0b51-48a0-b7b6-81edc7bff526\" (UID: \"94c7f5f6-0b51-48a0-b7b6-81edc7bff526\") "
Nov 03 18:01:21.740336 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:21.740307    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94c7f5f6-0b51-48a0-b7b6-81edc7bff526-maas-api-openshift-ca" pod "94c7f5f6-0b51-48a0-b7b6-81edc7bff526" (UID: "94c7f5f6-0b51-48a0-b7b6-81edc7bff526"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:01:21.740473 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:21.740452    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94c7f5f6-0b51-48a0-b7b6-81edc7bff526-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:01:21.742225 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:21.742205    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94c7f5f6-0b51-48a0-b7b6-81edc7bff526-maas-api-xks-ca" pod "94c7f5f6-0b51-48a0-b7b6-81edc7bff526" (UID: "94c7f5f6-0b51-48a0-b7b6-81edc7bff526"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:01:21.742319 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:21.742288    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94c7f5f6-0b51-48a0-b7b6-81edc7bff526-kube-api-access-dlqpj" pod "94c7f5f6-0b51-48a0-b7b6-81edc7bff526" (UID: "94c7f5f6-0b51-48a0-b7b6-81edc7bff526"). InnerVolumeSpecName "kube-api-access-dlqpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:01:21.841633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:21.841573    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94c7f5f6-0b51-48a0-b7b6-81edc7bff526-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:01:21.841633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:21.841605    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlqpj\" (UniqueName: \"kubernetes.io/projected/94c7f5f6-0b51-48a0-b7b6-81edc7bff526-kube-api-access-dlqpj\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:01:21.977595 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:21.977559    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr"]
Nov 03 18:01:21.978084 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:21.978069    5315 cpu_manager.go:414] "containerMap: removing container" podUID="94c7f5f6-0b51-48a0-b7b6-81edc7bff526" containerName="revoke-keys"
Nov 03 18:01:21.978084 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:21.978084    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94c7f5f6-0b51-48a0-b7b6-81edc7bff526" containerName="revoke-keys"
Nov 03 18:01:21.978175 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:21.978165    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="94c7f5f6-0b51-48a0-b7b6-81edc7bff526" containerName="revoke-keys"
Nov 03 18:01:21.982987 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:21.982964    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr"
Nov 03 18:01:22.053128 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.053094    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-r2jgr\" (UID: \"8f5bff5e-4b15-4cde-a3b0-71c9ae666122\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr"
Nov 03 18:01:22.053266 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.053158    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbl24\" (UniqueName: \"kubernetes.io/projected/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-kube-api-access-xbl24\") pod \"maas-default-gateway-openshift-default-6dcf757899-r2jgr\" (UID: \"8f5bff5e-4b15-4cde-a3b0-71c9ae666122\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr"
Nov 03 18:01:22.053266 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.053241    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-r2jgr\" (UID: \"8f5bff5e-4b15-4cde-a3b0-71c9ae666122\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr"
Nov 03 18:01:22.053348 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.053286    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-r2jgr\" (UID: \"8f5bff5e-4b15-4cde-a3b0-71c9ae666122\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr"
Nov 03 18:01:22.053348 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.053335    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-r2jgr\" (UID: \"8f5bff5e-4b15-4cde-a3b0-71c9ae666122\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr"
Nov 03 18:01:22.053419 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.053402    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-r2jgr\" (UID: \"8f5bff5e-4b15-4cde-a3b0-71c9ae666122\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr"
Nov 03 18:01:22.053454 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.053425    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-r2jgr\" (UID: \"8f5bff5e-4b15-4cde-a3b0-71c9ae666122\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr"
Nov 03 18:01:22.053454 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.053450    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-r2jgr\" (UID: \"8f5bff5e-4b15-4cde-a3b0-71c9ae666122\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr"
Nov 03 18:01:22.053523 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.053467    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-r2jgr\" (UID: \"8f5bff5e-4b15-4cde-a3b0-71c9ae666122\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr"
Nov 03 18:01:22.073218 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.073165    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr"]
Nov 03 18:01:22.154801 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.154764    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbl24\" (UniqueName: \"kubernetes.io/projected/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-kube-api-access-xbl24\") pod \"maas-default-gateway-openshift-default-6dcf757899-r2jgr\" (UID: \"8f5bff5e-4b15-4cde-a3b0-71c9ae666122\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr"
Nov 03 18:01:22.154982 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.154811    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-r2jgr\" (UID: \"8f5bff5e-4b15-4cde-a3b0-71c9ae666122\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr"
Nov 03 18:01:22.154982 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.154840    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-r2jgr\" (UID: \"8f5bff5e-4b15-4cde-a3b0-71c9ae666122\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr"
Nov 03 18:01:22.154982 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.154874    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-r2jgr\" (UID: \"8f5bff5e-4b15-4cde-a3b0-71c9ae666122\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr"
Nov 03 18:01:22.154982 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.154930    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-r2jgr\" (UID: \"8f5bff5e-4b15-4cde-a3b0-71c9ae666122\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr"
Nov 03 18:01:22.154982 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.154963    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-r2jgr\" (UID: \"8f5bff5e-4b15-4cde-a3b0-71c9ae666122\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr"
Nov 03 18:01:22.155241 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.155000    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-r2jgr\" (UID: \"8f5bff5e-4b15-4cde-a3b0-71c9ae666122\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr"
Nov 03 18:01:22.155241 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.155025    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-r2jgr\" (UID: \"8f5bff5e-4b15-4cde-a3b0-71c9ae666122\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr"
Nov 03 18:01:22.155241 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.155058    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-r2jgr\" (UID: \"8f5bff5e-4b15-4cde-a3b0-71c9ae666122\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr"
Nov 03 18:01:22.155401 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.155289    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-r2jgr\" (UID: \"8f5bff5e-4b15-4cde-a3b0-71c9ae666122\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr"
Nov 03 18:01:22.155401 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.155308    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-r2jgr\" (UID: \"8f5bff5e-4b15-4cde-a3b0-71c9ae666122\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr"
Nov 03 18:01:22.155401 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.155354    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-r2jgr\" (UID: \"8f5bff5e-4b15-4cde-a3b0-71c9ae666122\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr"
Nov 03 18:01:22.155556 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.155537    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-r2jgr\" (UID: \"8f5bff5e-4b15-4cde-a3b0-71c9ae666122\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr"
Nov 03 18:01:22.155785 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.155762    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-r2jgr\" (UID: \"8f5bff5e-4b15-4cde-a3b0-71c9ae666122\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr"
Nov 03 18:01:22.157708 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.157690    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-r2jgr\" (UID: \"8f5bff5e-4b15-4cde-a3b0-71c9ae666122\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr"
Nov 03 18:01:22.157923 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.157903    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-r2jgr\" (UID: \"8f5bff5e-4b15-4cde-a3b0-71c9ae666122\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr"
Nov 03 18:01:22.214667 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.214564    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbl24\" (UniqueName: \"kubernetes.io/projected/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-kube-api-access-xbl24\") pod \"maas-default-gateway-openshift-default-6dcf757899-r2jgr\" (UID: \"8f5bff5e-4b15-4cde-a3b0-71c9ae666122\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr"
Nov 03 18:01:22.214667 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.214591    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-r2jgr\" (UID: \"8f5bff5e-4b15-4cde-a3b0-71c9ae666122\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr"
Nov 03 18:01:22.277288 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.277237    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00110e0a-hj978"]
Nov 03 18:01:22.293591 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.293554    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00110e0a-hj978"
Nov 03 18:01:22.296306 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.295863    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr"
Nov 03 18:01:22.349769 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.349702    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00110e0a-hj978"]
Nov 03 18:01:22.360145 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.357537    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/581376ba-0009-4555-8999-ba410f3eb7e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00110e0a-hj978\" (UID: \"581376ba-0009-4555-8999-ba410f3eb7e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00110e0a-hj978"
Nov 03 18:01:22.360145 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.357695    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/581376ba-0009-4555-8999-ba410f3eb7e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00110e0a-hj978\" (UID: \"581376ba-0009-4555-8999-ba410f3eb7e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00110e0a-hj978"
Nov 03 18:01:22.360145 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.358580    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpf2r\" (UniqueName: \"kubernetes.io/projected/581376ba-0009-4555-8999-ba410f3eb7e6-kube-api-access-bpf2r\") pod \"maas-api-revoke-sub-00110e0a-hj978\" (UID: \"581376ba-0009-4555-8999-ba410f3eb7e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00110e0a-hj978"
Nov 03 18:01:22.460156 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.460115    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/581376ba-0009-4555-8999-ba410f3eb7e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00110e0a-hj978\" (UID: \"581376ba-0009-4555-8999-ba410f3eb7e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00110e0a-hj978"
Nov 03 18:01:22.460337 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.460167    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpf2r\" (UniqueName: \"kubernetes.io/projected/581376ba-0009-4555-8999-ba410f3eb7e6-kube-api-access-bpf2r\") pod \"maas-api-revoke-sub-00110e0a-hj978\" (UID: \"581376ba-0009-4555-8999-ba410f3eb7e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00110e0a-hj978"
Nov 03 18:01:22.460337 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.460224    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/581376ba-0009-4555-8999-ba410f3eb7e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00110e0a-hj978\" (UID: \"581376ba-0009-4555-8999-ba410f3eb7e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00110e0a-hj978"
Nov 03 18:01:22.460902 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.460875    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/581376ba-0009-4555-8999-ba410f3eb7e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00110e0a-hj978\" (UID: \"581376ba-0009-4555-8999-ba410f3eb7e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00110e0a-hj978"
Nov 03 18:01:22.462563 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.462535    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/581376ba-0009-4555-8999-ba410f3eb7e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00110e0a-hj978\" (UID: \"581376ba-0009-4555-8999-ba410f3eb7e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00110e0a-hj978"
Nov 03 18:01:22.517723 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.517647    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpf2r\" (UniqueName: \"kubernetes.io/projected/581376ba-0009-4555-8999-ba410f3eb7e6-kube-api-access-bpf2r\") pod \"maas-api-revoke-sub-00110e0a-hj978\" (UID: \"581376ba-0009-4555-8999-ba410f3eb7e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00110e0a-hj978"
Nov 03 18:01:22.546903 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.546877    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f88012a-7nkk7"
Nov 03 18:01:22.547081 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.546868    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f88012a-7nkk7" event={"ID":"94c7f5f6-0b51-48a0-b7b6-81edc7bff526","Type":"ContainerDied","Data":"d43be26cccc7823b8befa125bf6ec5a838bf91d2311a0461aa1cb3a2fde03b1b"}
Nov 03 18:01:22.547081 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.547048    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d43be26cccc7823b8befa125bf6ec5a838bf91d2311a0461aa1cb3a2fde03b1b"
Nov 03 18:01:22.548606 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.548580    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e91a8237-vvgc5" event={"ID":"f2943536-963c-462e-9715-7756e5af31f6","Type":"ContainerStarted","Data":"c494d0b3b02155e3f21621c1c576ca19ed8f5748941568380adbc8c46d4504bc"}
Nov 03 18:01:22.548747 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.548641    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e91a8237-vvgc5" event={"ID":"f2943536-963c-462e-9715-7756e5af31f6","Type":"ContainerStarted","Data":"a212ec5eb7251288504295ea27b79f26e66125117e4065a0cc2819e016d519cc"}
Nov 03 18:01:22.619080 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.619043    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00110e0a-hj978"
Nov 03 18:01:22.658202 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.658146    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e91a8237-vvgc5" podStartSLOduration=2.658131638 podStartE2EDuration="2.658131638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:01:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:01:22.656820862 +0000 UTC m=+921.673018932" watchObservedRunningTime="2026-11-03 18:01:22.658131638 +0000 UTC m=+921.674329707"
Nov 03 18:01:22.688092 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.687833    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr"]
Nov 03 18:01:22.694408 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:01:22.694373    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f5bff5e_4b15_4cde_a3b0_71c9ae666122.slice/crio-e204e645d8cbbe6f47ef145907e613050a99e12e29b66f5427019eabe09d8169 WatchSource:0}: Error finding container e204e645d8cbbe6f47ef145907e613050a99e12e29b66f5427019eabe09d8169: Status 404 returned error can't find the container with id e204e645d8cbbe6f47ef145907e613050a99e12e29b66f5427019eabe09d8169
Nov 03 18:01:22.697332 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.697297    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:01:22.697471 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.697367    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:01:22.697471 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.697396    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:01:22.991403 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:22.991371    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00110e0a-hj978"]
Nov 03 18:01:22.996899 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:01:22.996865    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod581376ba_0009_4555_8999_ba410f3eb7e6.slice/crio-2d3d4dc85e439b0ffc23850c2d60afbcc9a08dd2a386d9adb97af0db96a1c211 WatchSource:0}: Error finding container 2d3d4dc85e439b0ffc23850c2d60afbcc9a08dd2a386d9adb97af0db96a1c211: Status 404 returned error can't find the container with id 2d3d4dc85e439b0ffc23850c2d60afbcc9a08dd2a386d9adb97af0db96a1c211
Nov 03 18:01:23.557095 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:23.557055    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr" event={"ID":"8f5bff5e-4b15-4cde-a3b0-71c9ae666122","Type":"ContainerStarted","Data":"61bcac3edd0b11569024e70049f166240dc379a0b07b8d6febfe7be692b1e5f7"}
Nov 03 18:01:23.557285 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:23.557118    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr" event={"ID":"8f5bff5e-4b15-4cde-a3b0-71c9ae666122","Type":"ContainerStarted","Data":"e204e645d8cbbe6f47ef145907e613050a99e12e29b66f5427019eabe09d8169"}
Nov 03 18:01:23.558794 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:23.558769    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00110e0a-hj978" event={"ID":"581376ba-0009-4555-8999-ba410f3eb7e6","Type":"ContainerStarted","Data":"03e894e4d837336e8c020126a0448f44dba33db43979559a0977a4c59f2c3d95"}
Nov 03 18:01:23.558952 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:23.558814    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00110e0a-hj978" event={"ID":"581376ba-0009-4555-8999-ba410f3eb7e6","Type":"ContainerStarted","Data":"2d3d4dc85e439b0ffc23850c2d60afbcc9a08dd2a386d9adb97af0db96a1c211"}
Nov 03 18:01:23.561224 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:23.561196    5315 generic.go:356] "Generic (PLEG): container finished" podID="f2943536-963c-462e-9715-7756e5af31f6" containerID="c494d0b3b02155e3f21621c1c576ca19ed8f5748941568380adbc8c46d4504bc" exitCode=0
Nov 03 18:01:23.561339 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:23.561272    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e91a8237-vvgc5" event={"ID":"f2943536-963c-462e-9715-7756e5af31f6","Type":"ContainerDied","Data":"c494d0b3b02155e3f21621c1c576ca19ed8f5748941568380adbc8c46d4504bc"}
Nov 03 18:01:23.674536 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:23.674484    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr" podStartSLOduration=2.674472182 podStartE2EDuration="2.674472182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:01:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:01:23.672208204 +0000 UTC m=+922.688406273" watchObservedRunningTime="2026-11-03 18:01:23.674472182 +0000 UTC m=+922.690670250"
Nov 03 18:01:23.895766 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:23.895715    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00110e0a-hj978" podStartSLOduration=1.8957011609999999 podStartE2EDuration="1.895701161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:01:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:01:23.894036758 +0000 UTC m=+922.910234828" watchObservedRunningTime="2026-11-03 18:01:23.895701161 +0000 UTC m=+922.911899230"
Nov 03 18:01:24.297893 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:24.297847    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr"
Nov 03 18:01:24.299359 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:24.299330    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-r2jgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 18:01:24.299511 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:24.299381    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr" podUID="8f5bff5e-4b15-4cde-a3b0-71c9ae666122" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 18:01:24.569712 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:24.569602    5315 generic.go:356] "Generic (PLEG): container finished" podID="581376ba-0009-4555-8999-ba410f3eb7e6" containerID="03e894e4d837336e8c020126a0448f44dba33db43979559a0977a4c59f2c3d95" exitCode=0
Nov 03 18:01:24.569712 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:24.569668    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00110e0a-hj978" event={"ID":"581376ba-0009-4555-8999-ba410f3eb7e6","Type":"ContainerDied","Data":"03e894e4d837336e8c020126a0448f44dba33db43979559a0977a4c59f2c3d95"}
Nov 03 18:01:24.581628 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:01:24.581579    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:01:24.783659 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:24.783629    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e91a8237-vvgc5"
Nov 03 18:01:24.884630 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:24.884516    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2943536-963c-462e-9715-7756e5af31f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2943536-963c-462e-9715-7756e5af31f6-maas-api-openshift-ca\") pod \"f2943536-963c-462e-9715-7756e5af31f6\" (UID: \"f2943536-963c-462e-9715-7756e5af31f6\") "
Nov 03 18:01:24.884786 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:24.884730    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2943536-963c-462e-9715-7756e5af31f6-kube-api-access-x8qkt\" (UniqueName: \"kubernetes.io/projected/f2943536-963c-462e-9715-7756e5af31f6-kube-api-access-x8qkt\") pod \"f2943536-963c-462e-9715-7756e5af31f6\" (UID: \"f2943536-963c-462e-9715-7756e5af31f6\") "
Nov 03 18:01:24.884786 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:24.884766    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2943536-963c-462e-9715-7756e5af31f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2943536-963c-462e-9715-7756e5af31f6-maas-api-xks-ca\") pod \"f2943536-963c-462e-9715-7756e5af31f6\" (UID: \"f2943536-963c-462e-9715-7756e5af31f6\") "
Nov 03 18:01:24.884903 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:24.884864    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2943536-963c-462e-9715-7756e5af31f6-maas-api-openshift-ca" pod "f2943536-963c-462e-9715-7756e5af31f6" (UID: "f2943536-963c-462e-9715-7756e5af31f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:01:24.885123 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:24.885105    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2943536-963c-462e-9715-7756e5af31f6-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:01:24.886821 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:24.886798    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2943536-963c-462e-9715-7756e5af31f6-kube-api-access-x8qkt" pod "f2943536-963c-462e-9715-7756e5af31f6" (UID: "f2943536-963c-462e-9715-7756e5af31f6"). InnerVolumeSpecName "kube-api-access-x8qkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:01:24.886935 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:24.886915    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2943536-963c-462e-9715-7756e5af31f6-maas-api-xks-ca" pod "f2943536-963c-462e-9715-7756e5af31f6" (UID: "f2943536-963c-462e-9715-7756e5af31f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:01:24.986444 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:24.986404    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8qkt\" (UniqueName: \"kubernetes.io/projected/f2943536-963c-462e-9715-7756e5af31f6-kube-api-access-x8qkt\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:01:24.986444 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:24.986439    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2943536-963c-462e-9715-7756e5af31f6-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:01:25.297715 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:25.297675    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-r2jgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 18:01:25.297888 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:25.297754    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr" podUID="8f5bff5e-4b15-4cde-a3b0-71c9ae666122" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 18:01:25.576814 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:25.576730    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e91a8237-vvgc5"
Nov 03 18:01:25.576814 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:25.576758    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e91a8237-vvgc5" event={"ID":"f2943536-963c-462e-9715-7756e5af31f6","Type":"ContainerDied","Data":"a212ec5eb7251288504295ea27b79f26e66125117e4065a0cc2819e016d519cc"}
Nov 03 18:01:25.576814 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:25.576811    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a212ec5eb7251288504295ea27b79f26e66125117e4065a0cc2819e016d519cc"
Nov 03 18:01:25.757246 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:25.757217    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00110e0a-hj978"
Nov 03 18:01:25.794564 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:25.794532    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/581376ba-0009-4555-8999-ba410f3eb7e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/581376ba-0009-4555-8999-ba410f3eb7e6-maas-api-openshift-ca\") pod \"581376ba-0009-4555-8999-ba410f3eb7e6\" (UID: \"581376ba-0009-4555-8999-ba410f3eb7e6\") "
Nov 03 18:01:25.794741 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:25.794636    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/581376ba-0009-4555-8999-ba410f3eb7e6-kube-api-access-bpf2r\" (UniqueName: \"kubernetes.io/projected/581376ba-0009-4555-8999-ba410f3eb7e6-kube-api-access-bpf2r\") pod \"581376ba-0009-4555-8999-ba410f3eb7e6\" (UID: \"581376ba-0009-4555-8999-ba410f3eb7e6\") "
Nov 03 18:01:25.794741 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:25.794698    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/581376ba-0009-4555-8999-ba410f3eb7e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/581376ba-0009-4555-8999-ba410f3eb7e6-maas-api-xks-ca\") pod \"581376ba-0009-4555-8999-ba410f3eb7e6\" (UID: \"581376ba-0009-4555-8999-ba410f3eb7e6\") "
Nov 03 18:01:25.794921 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:25.794896    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/581376ba-0009-4555-8999-ba410f3eb7e6-maas-api-openshift-ca" pod "581376ba-0009-4555-8999-ba410f3eb7e6" (UID: "581376ba-0009-4555-8999-ba410f3eb7e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:01:25.795046 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:25.795031    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/581376ba-0009-4555-8999-ba410f3eb7e6-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:01:25.796764 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:25.796738    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/581376ba-0009-4555-8999-ba410f3eb7e6-maas-api-xks-ca" pod "581376ba-0009-4555-8999-ba410f3eb7e6" (UID: "581376ba-0009-4555-8999-ba410f3eb7e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:01:25.796864 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:25.796845    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/581376ba-0009-4555-8999-ba410f3eb7e6-kube-api-access-bpf2r" pod "581376ba-0009-4555-8999-ba410f3eb7e6" (UID: "581376ba-0009-4555-8999-ba410f3eb7e6"). InnerVolumeSpecName "kube-api-access-bpf2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:01:25.896684 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:25.896591    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/581376ba-0009-4555-8999-ba410f3eb7e6-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:01:25.896684 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:25.896636    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpf2r\" (UniqueName: \"kubernetes.io/projected/581376ba-0009-4555-8999-ba410f3eb7e6-kube-api-access-bpf2r\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:01:26.297701 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:26.297671    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-r2jgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 18:01:26.298097 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:26.297740    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr" podUID="8f5bff5e-4b15-4cde-a3b0-71c9ae666122" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 18:01:26.585884 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:26.585786    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00110e0a-hj978"
Nov 03 18:01:26.585884 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:26.585814    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00110e0a-hj978" event={"ID":"581376ba-0009-4555-8999-ba410f3eb7e6","Type":"ContainerDied","Data":"2d3d4dc85e439b0ffc23850c2d60afbcc9a08dd2a386d9adb97af0db96a1c211"}
Nov 03 18:01:26.585884 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:26.585878    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d3d4dc85e439b0ffc23850c2d60afbcc9a08dd2a386d9adb97af0db96a1c211"
Nov 03 18:01:27.297924 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:27.297891    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-r2jgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 18:01:27.298335 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:27.297945    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr" podUID="8f5bff5e-4b15-4cde-a3b0-71c9ae666122" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 18:01:28.297438 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:28.297397    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-r2jgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 18:01:28.297639 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:28.297458    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr" podUID="8f5bff5e-4b15-4cde-a3b0-71c9ae666122" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 18:01:29.297678 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:29.297642    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-r2jgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 18:01:29.298106 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:29.297706    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr" podUID="8f5bff5e-4b15-4cde-a3b0-71c9ae666122" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 18:01:31.318984 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:31.318943    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-r2jgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:01:31.319677 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:31.319641    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr" podUID="8f5bff5e-4b15-4cde-a3b0-71c9ae666122" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:01:31.452638 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:31.449016    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr"
Nov 03 18:01:31.452638 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:31.450973    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr"
Nov 03 18:01:31.452638 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:31.452219    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr"
Nov 03 18:01:32.223600 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:32.223561    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bea68c5-m9c4r"]
Nov 03 18:01:32.224322 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:32.224298    5315 cpu_manager.go:414] "containerMap: removing container" podUID="f2943536-963c-462e-9715-7756e5af31f6" containerName="revoke-keys"
Nov 03 18:01:32.224375 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:32.224328    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2943536-963c-462e-9715-7756e5af31f6" containerName="revoke-keys"
Nov 03 18:01:32.224415 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:32.224373    5315 cpu_manager.go:414] "containerMap: removing container" podUID="581376ba-0009-4555-8999-ba410f3eb7e6" containerName="revoke-keys"
Nov 03 18:01:32.224415 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:32.224383    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="581376ba-0009-4555-8999-ba410f3eb7e6" containerName="revoke-keys"
Nov 03 18:01:32.224516 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:32.224495    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="581376ba-0009-4555-8999-ba410f3eb7e6" containerName="revoke-keys"
Nov 03 18:01:32.224589 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:32.224576    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2943536-963c-462e-9715-7756e5af31f6" containerName="revoke-keys"
Nov 03 18:01:32.228986 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:32.228961    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bea68c5-m9c4r"
Nov 03 18:01:32.279190 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:32.279149    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:01:32.279446 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:32.279156    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:01:32.279598 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:32.279250    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:01:32.279598 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:32.279312    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:01:32.288626 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:32.288584    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bea68c5-m9c4r"]
Nov 03 18:01:32.333352 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:32.333316    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqkpp\" (UniqueName: \"kubernetes.io/projected/85c0fe57-c156-4fb6-a72f-e3faabada5d9-kube-api-access-bqkpp\") pod \"maas-api-revoke-sub-5bea68c5-m9c4r\" (UID: \"85c0fe57-c156-4fb6-a72f-e3faabada5d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bea68c5-m9c4r"
Nov 03 18:01:32.333875 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:32.333402    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85c0fe57-c156-4fb6-a72f-e3faabada5d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bea68c5-m9c4r\" (UID: \"85c0fe57-c156-4fb6-a72f-e3faabada5d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bea68c5-m9c4r"
Nov 03 18:01:32.333875 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:32.333479    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85c0fe57-c156-4fb6-a72f-e3faabada5d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bea68c5-m9c4r\" (UID: \"85c0fe57-c156-4fb6-a72f-e3faabada5d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bea68c5-m9c4r"
Nov 03 18:01:32.434430 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:32.434373    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqkpp\" (UniqueName: \"kubernetes.io/projected/85c0fe57-c156-4fb6-a72f-e3faabada5d9-kube-api-access-bqkpp\") pod \"maas-api-revoke-sub-5bea68c5-m9c4r\" (UID: \"85c0fe57-c156-4fb6-a72f-e3faabada5d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bea68c5-m9c4r"
Nov 03 18:01:32.434639 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:32.434504    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85c0fe57-c156-4fb6-a72f-e3faabada5d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bea68c5-m9c4r\" (UID: \"85c0fe57-c156-4fb6-a72f-e3faabada5d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bea68c5-m9c4r"
Nov 03 18:01:32.434639 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:32.434631    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85c0fe57-c156-4fb6-a72f-e3faabada5d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bea68c5-m9c4r\" (UID: \"85c0fe57-c156-4fb6-a72f-e3faabada5d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bea68c5-m9c4r"
Nov 03 18:01:32.435347 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:32.435321    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85c0fe57-c156-4fb6-a72f-e3faabada5d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bea68c5-m9c4r\" (UID: \"85c0fe57-c156-4fb6-a72f-e3faabada5d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bea68c5-m9c4r"
Nov 03 18:01:32.437145 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:32.437122    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85c0fe57-c156-4fb6-a72f-e3faabada5d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bea68c5-m9c4r\" (UID: \"85c0fe57-c156-4fb6-a72f-e3faabada5d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bea68c5-m9c4r"
Nov 03 18:01:32.492204 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:32.492133    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqkpp\" (UniqueName: \"kubernetes.io/projected/85c0fe57-c156-4fb6-a72f-e3faabada5d9-kube-api-access-bqkpp\") pod \"maas-api-revoke-sub-5bea68c5-m9c4r\" (UID: \"85c0fe57-c156-4fb6-a72f-e3faabada5d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bea68c5-m9c4r"
Nov 03 18:01:32.543363 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:32.543331    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bea68c5-m9c4r"
Nov 03 18:01:32.921891 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:32.921852    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bea68c5-m9c4r"]
Nov 03 18:01:32.927703 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:01:32.927656    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85c0fe57_c156_4fb6_a72f_e3faabada5d9.slice/crio-99e990510acddfa54c9b4ff33f26751f8d880b48e464cd800307190a9b242df3 WatchSource:0}: Error finding container 99e990510acddfa54c9b4ff33f26751f8d880b48e464cd800307190a9b242df3: Status 404 returned error can't find the container with id 99e990510acddfa54c9b4ff33f26751f8d880b48e464cd800307190a9b242df3
Nov 03 18:01:33.664063 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:33.664020    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bea68c5-m9c4r" event={"ID":"85c0fe57-c156-4fb6-a72f-e3faabada5d9","Type":"ContainerStarted","Data":"0fd60b94e70fe236fc7ad1f80a53943d9255e5885ab2f2e3e4ab09fb64e20c3d"}
Nov 03 18:01:33.664448 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:33.664085    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bea68c5-m9c4r" event={"ID":"85c0fe57-c156-4fb6-a72f-e3faabada5d9","Type":"ContainerStarted","Data":"99e990510acddfa54c9b4ff33f26751f8d880b48e464cd800307190a9b242df3"}
Nov 03 18:01:33.779748 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:33.779695    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bea68c5-m9c4r" podStartSLOduration=1.779681472 podStartE2EDuration="1.779681472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:01:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:01:33.778636449 +0000 UTC m=+932.794834519" watchObservedRunningTime="2026-11-03 18:01:33.779681472 +0000 UTC m=+932.795879558"
Nov 03 18:01:34.672081 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:34.672046    5315 generic.go:356] "Generic (PLEG): container finished" podID="85c0fe57-c156-4fb6-a72f-e3faabada5d9" containerID="0fd60b94e70fe236fc7ad1f80a53943d9255e5885ab2f2e3e4ab09fb64e20c3d" exitCode=0
Nov 03 18:01:34.672462 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:34.672108    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bea68c5-m9c4r" event={"ID":"85c0fe57-c156-4fb6-a72f-e3faabada5d9","Type":"ContainerDied","Data":"0fd60b94e70fe236fc7ad1f80a53943d9255e5885ab2f2e3e4ab09fb64e20c3d"}
Nov 03 18:01:35.161525 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:35.161491    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30d44741-mvwv6"]
Nov 03 18:01:35.165518 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:35.165495    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30d44741-mvwv6"
Nov 03 18:01:35.225540 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:35.225506    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30d44741-mvwv6"]
Nov 03 18:01:35.264737 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:35.264699    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8eea994-0b05-4bad-8342-b9652610fe74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30d44741-mvwv6\" (UID: \"a8eea994-0b05-4bad-8342-b9652610fe74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30d44741-mvwv6"
Nov 03 18:01:35.264901 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:35.264827    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8eea994-0b05-4bad-8342-b9652610fe74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30d44741-mvwv6\" (UID: \"a8eea994-0b05-4bad-8342-b9652610fe74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30d44741-mvwv6"
Nov 03 18:01:35.264901 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:35.264875    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85nmw\" (UniqueName: \"kubernetes.io/projected/a8eea994-0b05-4bad-8342-b9652610fe74-kube-api-access-85nmw\") pod \"maas-api-revoke-sub-30d44741-mvwv6\" (UID: \"a8eea994-0b05-4bad-8342-b9652610fe74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30d44741-mvwv6"
Nov 03 18:01:35.365825 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:35.365792    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8eea994-0b05-4bad-8342-b9652610fe74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30d44741-mvwv6\" (UID: \"a8eea994-0b05-4bad-8342-b9652610fe74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30d44741-mvwv6"
Nov 03 18:01:35.366018 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:35.365888    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8eea994-0b05-4bad-8342-b9652610fe74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30d44741-mvwv6\" (UID: \"a8eea994-0b05-4bad-8342-b9652610fe74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30d44741-mvwv6"
Nov 03 18:01:35.366018 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:35.365932    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85nmw\" (UniqueName: \"kubernetes.io/projected/a8eea994-0b05-4bad-8342-b9652610fe74-kube-api-access-85nmw\") pod \"maas-api-revoke-sub-30d44741-mvwv6\" (UID: \"a8eea994-0b05-4bad-8342-b9652610fe74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30d44741-mvwv6"
Nov 03 18:01:35.367029 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:35.366993    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8eea994-0b05-4bad-8342-b9652610fe74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30d44741-mvwv6\" (UID: \"a8eea994-0b05-4bad-8342-b9652610fe74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30d44741-mvwv6"
Nov 03 18:01:35.368446 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:35.368420    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8eea994-0b05-4bad-8342-b9652610fe74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30d44741-mvwv6\" (UID: \"a8eea994-0b05-4bad-8342-b9652610fe74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30d44741-mvwv6"
Nov 03 18:01:35.423479 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:35.423410    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85nmw\" (UniqueName: \"kubernetes.io/projected/a8eea994-0b05-4bad-8342-b9652610fe74-kube-api-access-85nmw\") pod \"maas-api-revoke-sub-30d44741-mvwv6\" (UID: \"a8eea994-0b05-4bad-8342-b9652610fe74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30d44741-mvwv6"
Nov 03 18:01:35.478392 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:35.478357    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30d44741-mvwv6"
Nov 03 18:01:35.806231 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:35.806203    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bea68c5-m9c4r"
Nov 03 18:01:35.854859 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:35.854826    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30d44741-mvwv6"]
Nov 03 18:01:35.860247 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:01:35.860220    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8eea994_0b05_4bad_8342_b9652610fe74.slice/crio-7497b470c0f49ad4272fbc84a2a6bef25cb75c57698c9ce67bc2afe0a536a751 WatchSource:0}: Error finding container 7497b470c0f49ad4272fbc84a2a6bef25cb75c57698c9ce67bc2afe0a536a751: Status 404 returned error can't find the container with id 7497b470c0f49ad4272fbc84a2a6bef25cb75c57698c9ce67bc2afe0a536a751
Nov 03 18:01:35.870066 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:35.870044    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85c0fe57-c156-4fb6-a72f-e3faabada5d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85c0fe57-c156-4fb6-a72f-e3faabada5d9-maas-api-xks-ca\") pod \"85c0fe57-c156-4fb6-a72f-e3faabada5d9\" (UID: \"85c0fe57-c156-4fb6-a72f-e3faabada5d9\") "
Nov 03 18:01:35.870166 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:35.870131    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85c0fe57-c156-4fb6-a72f-e3faabada5d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85c0fe57-c156-4fb6-a72f-e3faabada5d9-maas-api-openshift-ca\") pod \"85c0fe57-c156-4fb6-a72f-e3faabada5d9\" (UID: \"85c0fe57-c156-4fb6-a72f-e3faabada5d9\") "
Nov 03 18:01:35.870220 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:35.870186    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85c0fe57-c156-4fb6-a72f-e3faabada5d9-kube-api-access-bqkpp\" (UniqueName: \"kubernetes.io/projected/85c0fe57-c156-4fb6-a72f-e3faabada5d9-kube-api-access-bqkpp\") pod \"85c0fe57-c156-4fb6-a72f-e3faabada5d9\" (UID: \"85c0fe57-c156-4fb6-a72f-e3faabada5d9\") "
Nov 03 18:01:35.870508 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:35.870482    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85c0fe57-c156-4fb6-a72f-e3faabada5d9-maas-api-openshift-ca" pod "85c0fe57-c156-4fb6-a72f-e3faabada5d9" (UID: "85c0fe57-c156-4fb6-a72f-e3faabada5d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:01:35.872046 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:35.872020    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85c0fe57-c156-4fb6-a72f-e3faabada5d9-maas-api-xks-ca" pod "85c0fe57-c156-4fb6-a72f-e3faabada5d9" (UID: "85c0fe57-c156-4fb6-a72f-e3faabada5d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:01:35.872261 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:35.872235    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85c0fe57-c156-4fb6-a72f-e3faabada5d9-kube-api-access-bqkpp" pod "85c0fe57-c156-4fb6-a72f-e3faabada5d9" (UID: "85c0fe57-c156-4fb6-a72f-e3faabada5d9"). InnerVolumeSpecName "kube-api-access-bqkpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:01:35.972279 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:35.972199    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85c0fe57-c156-4fb6-a72f-e3faabada5d9-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:01:35.972279 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:35.972236    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85c0fe57-c156-4fb6-a72f-e3faabada5d9-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:01:35.972279 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:35.972252    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqkpp\" (UniqueName: \"kubernetes.io/projected/85c0fe57-c156-4fb6-a72f-e3faabada5d9-kube-api-access-bqkpp\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:01:36.685636 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:36.685504    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30d44741-mvwv6" event={"ID":"a8eea994-0b05-4bad-8342-b9652610fe74","Type":"ContainerStarted","Data":"63315cec3c5329265b683589e4aca34519005de46bdd6367aa762e410e5caec2"}
Nov 03 18:01:36.685636 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:36.685567    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30d44741-mvwv6" event={"ID":"a8eea994-0b05-4bad-8342-b9652610fe74","Type":"ContainerStarted","Data":"7497b470c0f49ad4272fbc84a2a6bef25cb75c57698c9ce67bc2afe0a536a751"}
Nov 03 18:01:36.686868 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:36.686847    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bea68c5-m9c4r"
Nov 03 18:01:36.686999 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:36.686862    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bea68c5-m9c4r" event={"ID":"85c0fe57-c156-4fb6-a72f-e3faabada5d9","Type":"ContainerDied","Data":"99e990510acddfa54c9b4ff33f26751f8d880b48e464cd800307190a9b242df3"}
Nov 03 18:01:36.686999 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:36.686902    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99e990510acddfa54c9b4ff33f26751f8d880b48e464cd800307190a9b242df3"
Nov 03 18:01:36.797753 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:36.797694    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30d44741-mvwv6" podStartSLOduration=1.797680124 podStartE2EDuration="1.797680124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:01:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:01:36.795390519 +0000 UTC m=+935.811588590" watchObservedRunningTime="2026-11-03 18:01:36.797680124 +0000 UTC m=+935.813878192"
Nov 03 18:01:37.694870 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:37.694836    5315 generic.go:356] "Generic (PLEG): container finished" podID="a8eea994-0b05-4bad-8342-b9652610fe74" containerID="63315cec3c5329265b683589e4aca34519005de46bdd6367aa762e410e5caec2" exitCode=0
Nov 03 18:01:37.695306 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:37.694901    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30d44741-mvwv6" event={"ID":"a8eea994-0b05-4bad-8342-b9652610fe74","Type":"ContainerDied","Data":"63315cec3c5329265b683589e4aca34519005de46bdd6367aa762e410e5caec2"}
Nov 03 18:01:38.845506 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:38.845478    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30d44741-mvwv6"
Nov 03 18:01:38.902591 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:38.902553    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8eea994-0b05-4bad-8342-b9652610fe74-kube-api-access-85nmw\" (UniqueName: \"kubernetes.io/projected/a8eea994-0b05-4bad-8342-b9652610fe74-kube-api-access-85nmw\") pod \"a8eea994-0b05-4bad-8342-b9652610fe74\" (UID: \"a8eea994-0b05-4bad-8342-b9652610fe74\") "
Nov 03 18:01:38.902802 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:38.902744    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8eea994-0b05-4bad-8342-b9652610fe74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8eea994-0b05-4bad-8342-b9652610fe74-maas-api-openshift-ca\") pod \"a8eea994-0b05-4bad-8342-b9652610fe74\" (UID: \"a8eea994-0b05-4bad-8342-b9652610fe74\") "
Nov 03 18:01:38.902870 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:38.902841    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8eea994-0b05-4bad-8342-b9652610fe74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8eea994-0b05-4bad-8342-b9652610fe74-maas-api-xks-ca\") pod \"a8eea994-0b05-4bad-8342-b9652610fe74\" (UID: \"a8eea994-0b05-4bad-8342-b9652610fe74\") "
Nov 03 18:01:38.903098 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:38.903053    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8eea994-0b05-4bad-8342-b9652610fe74-maas-api-openshift-ca" pod "a8eea994-0b05-4bad-8342-b9652610fe74" (UID: "a8eea994-0b05-4bad-8342-b9652610fe74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:01:38.903269 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:38.903249    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8eea994-0b05-4bad-8342-b9652610fe74-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:01:38.905216 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:38.905187    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8eea994-0b05-4bad-8342-b9652610fe74-maas-api-xks-ca" pod "a8eea994-0b05-4bad-8342-b9652610fe74" (UID: "a8eea994-0b05-4bad-8342-b9652610fe74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:01:38.905216 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:38.905203    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8eea994-0b05-4bad-8342-b9652610fe74-kube-api-access-85nmw" pod "a8eea994-0b05-4bad-8342-b9652610fe74" (UID: "a8eea994-0b05-4bad-8342-b9652610fe74"). InnerVolumeSpecName "kube-api-access-85nmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:01:39.004777 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:39.004690    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85nmw\" (UniqueName: \"kubernetes.io/projected/a8eea994-0b05-4bad-8342-b9652610fe74-kube-api-access-85nmw\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:01:39.004777 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:39.004721    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8eea994-0b05-4bad-8342-b9652610fe74-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:01:39.711850 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:39.711820    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30d44741-mvwv6"
Nov 03 18:01:39.712026 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:39.711847    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30d44741-mvwv6" event={"ID":"a8eea994-0b05-4bad-8342-b9652610fe74","Type":"ContainerDied","Data":"7497b470c0f49ad4272fbc84a2a6bef25cb75c57698c9ce67bc2afe0a536a751"}
Nov 03 18:01:39.712026 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:39.711897    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7497b470c0f49ad4272fbc84a2a6bef25cb75c57698c9ce67bc2afe0a536a751"
Nov 03 18:01:40.802175 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:40.802131    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p"]
Nov 03 18:01:40.802768 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:40.802748    5315 cpu_manager.go:414] "containerMap: removing container" podUID="85c0fe57-c156-4fb6-a72f-e3faabada5d9" containerName="revoke-keys"
Nov 03 18:01:40.802768 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:40.802769    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85c0fe57-c156-4fb6-a72f-e3faabada5d9" containerName="revoke-keys"
Nov 03 18:01:40.802882 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:40.802787    5315 cpu_manager.go:414] "containerMap: removing container" podUID="a8eea994-0b05-4bad-8342-b9652610fe74" containerName="revoke-keys"
Nov 03 18:01:40.802882 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:40.802793    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8eea994-0b05-4bad-8342-b9652610fe74" containerName="revoke-keys"
Nov 03 18:01:40.802955 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:40.802905    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="85c0fe57-c156-4fb6-a72f-e3faabada5d9" containerName="revoke-keys"
Nov 03 18:01:40.802955 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:40.802914    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8eea994-0b05-4bad-8342-b9652610fe74" containerName="revoke-keys"
Nov 03 18:01:40.806474 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:40.806453    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p"
Nov 03 18:01:40.857204 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:40.857175    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-0ccbb9-openshift-default-dockercfg-8dgdl\""
Nov 03 18:01:40.905717 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:40.905685    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p"]
Nov 03 18:01:40.925544 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:40.925507    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79050b41-0f3b-4fc4-a8e8-50316359ac33-istiod-ca-cert\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p\" (UID: \"79050b41-0f3b-4fc4-a8e8-50316359ac33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p"
Nov 03 18:01:40.925544 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:40.925550    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79050b41-0f3b-4fc4-a8e8-50316359ac33-credential-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p\" (UID: \"79050b41-0f3b-4fc4-a8e8-50316359ac33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p"
Nov 03 18:01:40.925733 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:40.925622    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79050b41-0f3b-4fc4-a8e8-50316359ac33-workload-certs\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p\" (UID: \"79050b41-0f3b-4fc4-a8e8-50316359ac33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p"
Nov 03 18:01:40.925733 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:40.925653    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79050b41-0f3b-4fc4-a8e8-50316359ac33-istio-envoy\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p\" (UID: \"79050b41-0f3b-4fc4-a8e8-50316359ac33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p"
Nov 03 18:01:40.925733 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:40.925689    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79050b41-0f3b-4fc4-a8e8-50316359ac33-workload-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p\" (UID: \"79050b41-0f3b-4fc4-a8e8-50316359ac33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p"
Nov 03 18:01:40.925733 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:40.925719    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79050b41-0f3b-4fc4-a8e8-50316359ac33-istio-token\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p\" (UID: \"79050b41-0f3b-4fc4-a8e8-50316359ac33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p"
Nov 03 18:01:40.925882 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:40.925741    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79050b41-0f3b-4fc4-a8e8-50316359ac33-istio-podinfo\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p\" (UID: \"79050b41-0f3b-4fc4-a8e8-50316359ac33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p"
Nov 03 18:01:40.925882 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:40.925758    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q66b5\" (UniqueName: \"kubernetes.io/projected/79050b41-0f3b-4fc4-a8e8-50316359ac33-kube-api-access-q66b5\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p\" (UID: \"79050b41-0f3b-4fc4-a8e8-50316359ac33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p"
Nov 03 18:01:40.925882 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:40.925787    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79050b41-0f3b-4fc4-a8e8-50316359ac33-istio-data\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p\" (UID: \"79050b41-0f3b-4fc4-a8e8-50316359ac33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p"
Nov 03 18:01:41.026979 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:41.026943    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79050b41-0f3b-4fc4-a8e8-50316359ac33-istiod-ca-cert\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p\" (UID: \"79050b41-0f3b-4fc4-a8e8-50316359ac33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p"
Nov 03 18:01:41.026979 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:41.026981    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79050b41-0f3b-4fc4-a8e8-50316359ac33-credential-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p\" (UID: \"79050b41-0f3b-4fc4-a8e8-50316359ac33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p"
Nov 03 18:01:41.027203 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:41.027030    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79050b41-0f3b-4fc4-a8e8-50316359ac33-workload-certs\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p\" (UID: \"79050b41-0f3b-4fc4-a8e8-50316359ac33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p"
Nov 03 18:01:41.027203 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:41.027050    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79050b41-0f3b-4fc4-a8e8-50316359ac33-istio-envoy\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p\" (UID: \"79050b41-0f3b-4fc4-a8e8-50316359ac33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p"
Nov 03 18:01:41.027203 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:41.027079    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79050b41-0f3b-4fc4-a8e8-50316359ac33-workload-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p\" (UID: \"79050b41-0f3b-4fc4-a8e8-50316359ac33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p"
Nov 03 18:01:41.027203 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:41.027119    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79050b41-0f3b-4fc4-a8e8-50316359ac33-istio-token\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p\" (UID: \"79050b41-0f3b-4fc4-a8e8-50316359ac33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p"
Nov 03 18:01:41.027203 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:41.027153    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79050b41-0f3b-4fc4-a8e8-50316359ac33-istio-podinfo\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p\" (UID: \"79050b41-0f3b-4fc4-a8e8-50316359ac33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p"
Nov 03 18:01:41.027203 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:41.027177    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q66b5\" (UniqueName: \"kubernetes.io/projected/79050b41-0f3b-4fc4-a8e8-50316359ac33-kube-api-access-q66b5\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p\" (UID: \"79050b41-0f3b-4fc4-a8e8-50316359ac33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p"
Nov 03 18:01:41.027517 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:41.027220    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79050b41-0f3b-4fc4-a8e8-50316359ac33-istio-data\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p\" (UID: \"79050b41-0f3b-4fc4-a8e8-50316359ac33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p"
Nov 03 18:01:41.027580 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:41.027523    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79050b41-0f3b-4fc4-a8e8-50316359ac33-credential-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p\" (UID: \"79050b41-0f3b-4fc4-a8e8-50316359ac33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p"
Nov 03 18:01:41.027663 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:41.027573    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79050b41-0f3b-4fc4-a8e8-50316359ac33-istio-data\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p\" (UID: \"79050b41-0f3b-4fc4-a8e8-50316359ac33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p"
Nov 03 18:01:41.027663 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:41.027598    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79050b41-0f3b-4fc4-a8e8-50316359ac33-workload-certs\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p\" (UID: \"79050b41-0f3b-4fc4-a8e8-50316359ac33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p"
Nov 03 18:01:41.027833 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:41.027791    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79050b41-0f3b-4fc4-a8e8-50316359ac33-istiod-ca-cert\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p\" (UID: \"79050b41-0f3b-4fc4-a8e8-50316359ac33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p"
Nov 03 18:01:41.027944 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:41.027922    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79050b41-0f3b-4fc4-a8e8-50316359ac33-workload-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p\" (UID: \"79050b41-0f3b-4fc4-a8e8-50316359ac33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p"
Nov 03 18:01:41.029549 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:41.029523    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79050b41-0f3b-4fc4-a8e8-50316359ac33-istio-envoy\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p\" (UID: \"79050b41-0f3b-4fc4-a8e8-50316359ac33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p"
Nov 03 18:01:41.029905 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:41.029882    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79050b41-0f3b-4fc4-a8e8-50316359ac33-istio-podinfo\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p\" (UID: \"79050b41-0f3b-4fc4-a8e8-50316359ac33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p"
Nov 03 18:01:41.086861 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:41.086794    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79050b41-0f3b-4fc4-a8e8-50316359ac33-istio-token\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p\" (UID: \"79050b41-0f3b-4fc4-a8e8-50316359ac33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p"
Nov 03 18:01:41.087007 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:41.086957    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q66b5\" (UniqueName: \"kubernetes.io/projected/79050b41-0f3b-4fc4-a8e8-50316359ac33-kube-api-access-q66b5\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p\" (UID: \"79050b41-0f3b-4fc4-a8e8-50316359ac33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p"
Nov 03 18:01:41.121320 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:41.121291    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p"
Nov 03 18:01:41.559652 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:41.559581    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p"]
Nov 03 18:01:41.563528 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:01:41.563503    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79050b41_0f3b_4fc4_a8e8_50316359ac33.slice/crio-e82b7f630e2085cd3b199bd014ee74b728e7c96a216562d33940419fceb16b9e WatchSource:0}: Error finding container e82b7f630e2085cd3b199bd014ee74b728e7c96a216562d33940419fceb16b9e: Status 404 returned error can't find the container with id e82b7f630e2085cd3b199bd014ee74b728e7c96a216562d33940419fceb16b9e
Nov 03 18:01:41.565911 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:41.565872    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:01:41.566009 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:41.565944    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:01:41.566009 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:41.565974    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:01:41.728648 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:41.726818    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p" event={"ID":"79050b41-0f3b-4fc4-a8e8-50316359ac33","Type":"ContainerStarted","Data":"cafbf7926af78b8aff2e8c4409d8633c9a4efb7801f5442b4f66d96444e558aa"}
Nov 03 18:01:41.728648 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:41.726890    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p" event={"ID":"79050b41-0f3b-4fc4-a8e8-50316359ac33","Type":"ContainerStarted","Data":"e82b7f630e2085cd3b199bd014ee74b728e7c96a216562d33940419fceb16b9e"}
Nov 03 18:01:41.852979 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:41.852868    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p" podStartSLOduration=1.8528550830000001 podStartE2EDuration="1.852855083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:01:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:01:41.850542144 +0000 UTC m=+940.866740213" watchObservedRunningTime="2026-11-03 18:01:41.852855083 +0000 UTC m=+940.869053153"
Nov 03 18:01:42.122247 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:42.122150    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p"
Nov 03 18:01:42.123531 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:42.123503    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 18:01:42.123669 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:42.123552    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p" podUID="79050b41-0f3b-4fc4-a8e8-50316359ac33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 18:01:43.122326 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:43.122286    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 18:01:43.122761 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:43.122341    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p" podUID="79050b41-0f3b-4fc4-a8e8-50316359ac33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 18:01:44.134568 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:44.134528    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 18:01:44.135314 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:44.135275    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p" podUID="79050b41-0f3b-4fc4-a8e8-50316359ac33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 18:01:45.122377 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:45.122340    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 18:01:45.122533 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:45.122406    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p" podUID="79050b41-0f3b-4fc4-a8e8-50316359ac33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 18:01:46.121914 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:46.121871    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 18:01:46.122354 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:46.121929    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p" podUID="79050b41-0f3b-4fc4-a8e8-50316359ac33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 18:01:47.122030 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:47.121995    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 18:01:47.122428 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:47.122061    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p" podUID="79050b41-0f3b-4fc4-a8e8-50316359ac33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 18:01:47.623179 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:47.623142    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdf4f8f0-nr8cs"]
Nov 03 18:01:47.649917 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:47.649880    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf4f8f0-nr8cs"
Nov 03 18:01:47.690176 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:47.690145    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdf4f8f0-nr8cs"]
Nov 03 18:01:47.746973 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:47.746934    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:01:47.747285 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:47.747264    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:01:47.747569 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:47.747551    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:01:47.749632 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:47.747913    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:01:47.761690 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:47.761660    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx4qt\" (UniqueName: \"kubernetes.io/projected/863d7747-9354-4c32-92de-2d4c26148682-kube-api-access-sx4qt\") pod \"maas-api-revoke-sub-cdf4f8f0-nr8cs\" (UID: \"863d7747-9354-4c32-92de-2d4c26148682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf4f8f0-nr8cs"
Nov 03 18:01:47.761888 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:47.761871    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/863d7747-9354-4c32-92de-2d4c26148682-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdf4f8f0-nr8cs\" (UID: \"863d7747-9354-4c32-92de-2d4c26148682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf4f8f0-nr8cs"
Nov 03 18:01:47.761955 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:47.761910    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/863d7747-9354-4c32-92de-2d4c26148682-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdf4f8f0-nr8cs\" (UID: \"863d7747-9354-4c32-92de-2d4c26148682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf4f8f0-nr8cs"
Nov 03 18:01:47.863299 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:47.863261    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/863d7747-9354-4c32-92de-2d4c26148682-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdf4f8f0-nr8cs\" (UID: \"863d7747-9354-4c32-92de-2d4c26148682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf4f8f0-nr8cs"
Nov 03 18:01:47.863520 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:47.863319    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/863d7747-9354-4c32-92de-2d4c26148682-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdf4f8f0-nr8cs\" (UID: \"863d7747-9354-4c32-92de-2d4c26148682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf4f8f0-nr8cs"
Nov 03 18:01:47.863585 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:47.863537    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx4qt\" (UniqueName: \"kubernetes.io/projected/863d7747-9354-4c32-92de-2d4c26148682-kube-api-access-sx4qt\") pod \"maas-api-revoke-sub-cdf4f8f0-nr8cs\" (UID: \"863d7747-9354-4c32-92de-2d4c26148682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf4f8f0-nr8cs"
Nov 03 18:01:47.864133 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:47.864103    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/863d7747-9354-4c32-92de-2d4c26148682-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdf4f8f0-nr8cs\" (UID: \"863d7747-9354-4c32-92de-2d4c26148682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf4f8f0-nr8cs"
Nov 03 18:01:47.865821 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:47.865797    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/863d7747-9354-4c32-92de-2d4c26148682-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdf4f8f0-nr8cs\" (UID: \"863d7747-9354-4c32-92de-2d4c26148682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf4f8f0-nr8cs"
Nov 03 18:01:47.921921 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:47.921843    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx4qt\" (UniqueName: \"kubernetes.io/projected/863d7747-9354-4c32-92de-2d4c26148682-kube-api-access-sx4qt\") pod \"maas-api-revoke-sub-cdf4f8f0-nr8cs\" (UID: \"863d7747-9354-4c32-92de-2d4c26148682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf4f8f0-nr8cs"
Nov 03 18:01:47.963501 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:47.963469    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf4f8f0-nr8cs"
Nov 03 18:01:48.122204 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:48.122167    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 18:01:48.122587 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:48.122221    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p" podUID="79050b41-0f3b-4fc4-a8e8-50316359ac33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 18:01:48.346472 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:48.346434    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdf4f8f0-nr8cs"]
Nov 03 18:01:48.351934 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:01:48.351900    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod863d7747_9354_4c32_92de_2d4c26148682.slice/crio-752fd11057d983fae8e5af3d3c62c8187234c9c78630e9abde3ede37f1643bab WatchSource:0}: Error finding container 752fd11057d983fae8e5af3d3c62c8187234c9c78630e9abde3ede37f1643bab: Status 404 returned error can't find the container with id 752fd11057d983fae8e5af3d3c62c8187234c9c78630e9abde3ede37f1643bab
Nov 03 18:01:48.805231 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:48.805193    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf4f8f0-nr8cs" event={"ID":"863d7747-9354-4c32-92de-2d4c26148682","Type":"ContainerStarted","Data":"d0d1932ca54da64aee3d19de587b7082f2cb29e6520da6b61011d37222e164df"}
Nov 03 18:01:48.805383 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:48.805260    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf4f8f0-nr8cs" event={"ID":"863d7747-9354-4c32-92de-2d4c26148682","Type":"ContainerStarted","Data":"752fd11057d983fae8e5af3d3c62c8187234c9c78630e9abde3ede37f1643bab"}
Nov 03 18:01:48.915778 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:48.915676    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf4f8f0-nr8cs" podStartSLOduration=1.915662104 podStartE2EDuration="1.915662104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:01:48.91436017 +0000 UTC m=+947.930558245" watchObservedRunningTime="2026-11-03 18:01:48.915662104 +0000 UTC m=+947.931860172"
Nov 03 18:01:49.387820 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:49.387786    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p"
Nov 03 18:01:49.388248 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:49.388183    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p"
Nov 03 18:01:49.389625 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:49.389583    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p"
Nov 03 18:01:49.813344 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:49.813309    5315 generic.go:356] "Generic (PLEG): container finished" podID="863d7747-9354-4c32-92de-2d4c26148682" containerID="d0d1932ca54da64aee3d19de587b7082f2cb29e6520da6b61011d37222e164df" exitCode=0
Nov 03 18:01:49.813511 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:49.813365    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf4f8f0-nr8cs" event={"ID":"863d7747-9354-4c32-92de-2d4c26148682","Type":"ContainerDied","Data":"d0d1932ca54da64aee3d19de587b7082f2cb29e6520da6b61011d37222e164df"}
Nov 03 18:01:50.971334 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:50.971304    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf4f8f0-nr8cs"
Nov 03 18:01:51.100848 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:51.100744    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/863d7747-9354-4c32-92de-2d4c26148682-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/863d7747-9354-4c32-92de-2d4c26148682-maas-api-xks-ca\") pod \"863d7747-9354-4c32-92de-2d4c26148682\" (UID: \"863d7747-9354-4c32-92de-2d4c26148682\") "
Nov 03 18:01:51.101023 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:51.100892    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/863d7747-9354-4c32-92de-2d4c26148682-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/863d7747-9354-4c32-92de-2d4c26148682-maas-api-openshift-ca\") pod \"863d7747-9354-4c32-92de-2d4c26148682\" (UID: \"863d7747-9354-4c32-92de-2d4c26148682\") "
Nov 03 18:01:51.101023 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:51.100943    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/863d7747-9354-4c32-92de-2d4c26148682-kube-api-access-sx4qt\" (UniqueName: \"kubernetes.io/projected/863d7747-9354-4c32-92de-2d4c26148682-kube-api-access-sx4qt\") pod \"863d7747-9354-4c32-92de-2d4c26148682\" (UID: \"863d7747-9354-4c32-92de-2d4c26148682\") "
Nov 03 18:01:51.101550 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:51.101514    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/863d7747-9354-4c32-92de-2d4c26148682-maas-api-openshift-ca" pod "863d7747-9354-4c32-92de-2d4c26148682" (UID: "863d7747-9354-4c32-92de-2d4c26148682"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:01:51.103080 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:51.103052    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/863d7747-9354-4c32-92de-2d4c26148682-maas-api-xks-ca" pod "863d7747-9354-4c32-92de-2d4c26148682" (UID: "863d7747-9354-4c32-92de-2d4c26148682"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:01:51.103748 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:51.103717    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/863d7747-9354-4c32-92de-2d4c26148682-kube-api-access-sx4qt" pod "863d7747-9354-4c32-92de-2d4c26148682" (UID: "863d7747-9354-4c32-92de-2d4c26148682"). InnerVolumeSpecName "kube-api-access-sx4qt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:01:51.202408 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:51.202357    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/863d7747-9354-4c32-92de-2d4c26148682-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:01:51.202408 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:51.202396    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/863d7747-9354-4c32-92de-2d4c26148682-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:01:51.202408 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:51.202408    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx4qt\" (UniqueName: \"kubernetes.io/projected/863d7747-9354-4c32-92de-2d4c26148682-kube-api-access-sx4qt\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:01:51.827164 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:51.827130    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf4f8f0-nr8cs"
Nov 03 18:01:51.827343 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:51.827169    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf4f8f0-nr8cs" event={"ID":"863d7747-9354-4c32-92de-2d4c26148682","Type":"ContainerDied","Data":"752fd11057d983fae8e5af3d3c62c8187234c9c78630e9abde3ede37f1643bab"}
Nov 03 18:01:51.827343 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:51.827247    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="752fd11057d983fae8e5af3d3c62c8187234c9c78630e9abde3ede37f1643bab"
Nov 03 18:01:54.578783 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:01:54.578739    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:01:55.620330 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:55.620295    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74"]
Nov 03 18:01:55.620868 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:55.620838    5315 cpu_manager.go:414] "containerMap: removing container" podUID="863d7747-9354-4c32-92de-2d4c26148682" containerName="revoke-keys"
Nov 03 18:01:55.620868 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:55.620861    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="863d7747-9354-4c32-92de-2d4c26148682" containerName="revoke-keys"
Nov 03 18:01:55.621110 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:55.621074    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="863d7747-9354-4c32-92de-2d4c26148682" containerName="revoke-keys"
Nov 03 18:01:55.625733 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:55.625697    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74"
Nov 03 18:01:55.675150 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:55.675118    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-aab753-openshift-default-dockercfg-2fhq6\""
Nov 03 18:01:55.723924 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:55.723883    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74"]
Nov 03 18:01:55.751045 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:55.751005    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghrhb\" (UniqueName: \"kubernetes.io/projected/4eeee197-126d-4005-9745-e7aa8d36c574-kube-api-access-ghrhb\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74\" (UID: \"4eeee197-126d-4005-9745-e7aa8d36c574\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74"
Nov 03 18:01:55.751231 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:55.751069    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eeee197-126d-4005-9745-e7aa8d36c574-istio-envoy\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74\" (UID: \"4eeee197-126d-4005-9745-e7aa8d36c574\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74"
Nov 03 18:01:55.751231 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:55.751150    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4eeee197-126d-4005-9745-e7aa8d36c574-istio-token\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74\" (UID: \"4eeee197-126d-4005-9745-e7aa8d36c574\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74"
Nov 03 18:01:55.751231 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:55.751180    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eeee197-126d-4005-9745-e7aa8d36c574-istiod-ca-cert\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74\" (UID: \"4eeee197-126d-4005-9745-e7aa8d36c574\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74"
Nov 03 18:01:55.751350 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:55.751259    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eeee197-126d-4005-9745-e7aa8d36c574-credential-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74\" (UID: \"4eeee197-126d-4005-9745-e7aa8d36c574\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74"
Nov 03 18:01:55.751350 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:55.751303    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eeee197-126d-4005-9745-e7aa8d36c574-workload-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74\" (UID: \"4eeee197-126d-4005-9745-e7aa8d36c574\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74"
Nov 03 18:01:55.751489 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:55.751473    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eeee197-126d-4005-9745-e7aa8d36c574-istio-podinfo\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74\" (UID: \"4eeee197-126d-4005-9745-e7aa8d36c574\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74"
Nov 03 18:01:55.751548 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:55.751511    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eeee197-126d-4005-9745-e7aa8d36c574-workload-certs\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74\" (UID: \"4eeee197-126d-4005-9745-e7aa8d36c574\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74"
Nov 03 18:01:55.751594 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:55.751554    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eeee197-126d-4005-9745-e7aa8d36c574-istio-data\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74\" (UID: \"4eeee197-126d-4005-9745-e7aa8d36c574\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74"
Nov 03 18:01:55.853039 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:55.853001    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eeee197-126d-4005-9745-e7aa8d36c574-istio-podinfo\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74\" (UID: \"4eeee197-126d-4005-9745-e7aa8d36c574\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74"
Nov 03 18:01:55.853039 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:55.853046    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eeee197-126d-4005-9745-e7aa8d36c574-workload-certs\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74\" (UID: \"4eeee197-126d-4005-9745-e7aa8d36c574\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74"
Nov 03 18:01:55.853299 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:55.853063    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eeee197-126d-4005-9745-e7aa8d36c574-istio-data\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74\" (UID: \"4eeee197-126d-4005-9745-e7aa8d36c574\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74"
Nov 03 18:01:55.853299 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:55.853095    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghrhb\" (UniqueName: \"kubernetes.io/projected/4eeee197-126d-4005-9745-e7aa8d36c574-kube-api-access-ghrhb\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74\" (UID: \"4eeee197-126d-4005-9745-e7aa8d36c574\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74"
Nov 03 18:01:55.853299 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:55.853120    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eeee197-126d-4005-9745-e7aa8d36c574-istio-envoy\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74\" (UID: \"4eeee197-126d-4005-9745-e7aa8d36c574\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74"
Nov 03 18:01:55.853299 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:55.853140    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4eeee197-126d-4005-9745-e7aa8d36c574-istio-token\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74\" (UID: \"4eeee197-126d-4005-9745-e7aa8d36c574\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74"
Nov 03 18:01:55.853299 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:55.853207    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eeee197-126d-4005-9745-e7aa8d36c574-istiod-ca-cert\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74\" (UID: \"4eeee197-126d-4005-9745-e7aa8d36c574\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74"
Nov 03 18:01:55.853299 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:55.853294    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eeee197-126d-4005-9745-e7aa8d36c574-credential-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74\" (UID: \"4eeee197-126d-4005-9745-e7aa8d36c574\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74"
Nov 03 18:01:55.853602 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:55.853347    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eeee197-126d-4005-9745-e7aa8d36c574-workload-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74\" (UID: \"4eeee197-126d-4005-9745-e7aa8d36c574\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74"
Nov 03 18:01:55.853602 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:55.853531    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eeee197-126d-4005-9745-e7aa8d36c574-workload-certs\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74\" (UID: \"4eeee197-126d-4005-9745-e7aa8d36c574\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74"
Nov 03 18:01:55.853757 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:55.853645    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eeee197-126d-4005-9745-e7aa8d36c574-istio-data\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74\" (UID: \"4eeee197-126d-4005-9745-e7aa8d36c574\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74"
Nov 03 18:01:55.853823 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:55.853803    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eeee197-126d-4005-9745-e7aa8d36c574-workload-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74\" (UID: \"4eeee197-126d-4005-9745-e7aa8d36c574\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74"
Nov 03 18:01:55.853894 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:55.853866    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eeee197-126d-4005-9745-e7aa8d36c574-credential-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74\" (UID: \"4eeee197-126d-4005-9745-e7aa8d36c574\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74"
Nov 03 18:01:55.854051 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:55.854031    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eeee197-126d-4005-9745-e7aa8d36c574-istiod-ca-cert\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74\" (UID: \"4eeee197-126d-4005-9745-e7aa8d36c574\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74"
Nov 03 18:01:55.855452 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:55.855426    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eeee197-126d-4005-9745-e7aa8d36c574-istio-envoy\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74\" (UID: \"4eeee197-126d-4005-9745-e7aa8d36c574\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74"
Nov 03 18:01:55.855625 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:55.855595    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eeee197-126d-4005-9745-e7aa8d36c574-istio-podinfo\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74\" (UID: \"4eeee197-126d-4005-9745-e7aa8d36c574\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74"
Nov 03 18:01:55.911018 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:55.910936    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghrhb\" (UniqueName: \"kubernetes.io/projected/4eeee197-126d-4005-9745-e7aa8d36c574-kube-api-access-ghrhb\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74\" (UID: \"4eeee197-126d-4005-9745-e7aa8d36c574\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74"
Nov 03 18:01:55.911018 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:55.910951    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4eeee197-126d-4005-9745-e7aa8d36c574-istio-token\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74\" (UID: \"4eeee197-126d-4005-9745-e7aa8d36c574\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74"
Nov 03 18:01:55.938884 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:55.938835    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74"
Nov 03 18:01:56.380361 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:56.380314    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74"]
Nov 03 18:01:56.413460 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:01:56.413417    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eeee197_126d_4005_9745_e7aa8d36c574.slice/crio-4ba33d75a9c7fb05c4bf5229412b42c0fba20ffab0e02e3651de4bf486fe0c56 WatchSource:0}: Error finding container 4ba33d75a9c7fb05c4bf5229412b42c0fba20ffab0e02e3651de4bf486fe0c56: Status 404 returned error can't find the container with id 4ba33d75a9c7fb05c4bf5229412b42c0fba20ffab0e02e3651de4bf486fe0c56
Nov 03 18:01:56.416201 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:56.416156    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:01:56.416280 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:56.416256    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:01:56.416320 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:56.416300    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:01:56.874652 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:56.873340    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74" event={"ID":"4eeee197-126d-4005-9745-e7aa8d36c574","Type":"ContainerStarted","Data":"176213a03e8120e0e463d61c18a8c649820b7e5370ddcfb8bc38714f237f1d34"}
Nov 03 18:01:56.874652 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:56.873472    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74" event={"ID":"4eeee197-126d-4005-9745-e7aa8d36c574","Type":"ContainerStarted","Data":"4ba33d75a9c7fb05c4bf5229412b42c0fba20ffab0e02e3651de4bf486fe0c56"}
Nov 03 18:01:56.943523 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:56.943480    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74"
Nov 03 18:01:57.004427 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:57.004370    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74" podStartSLOduration=2.00435577 podStartE2EDuration="2.00435577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:01:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:01:57.000580637 +0000 UTC m=+956.016778720" watchObservedRunningTime="2026-11-03 18:01:57.00435577 +0000 UTC m=+956.020553838"
Nov 03 18:01:57.947370 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:57.947329    5315 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 18:01:57.947985 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:57.947951    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74" podUID="4eeee197-126d-4005-9745-e7aa8d36c574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 18:01:58.949173 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:58.949133    5315 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 18:01:58.949824 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:58.949783    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74" podUID="4eeee197-126d-4005-9745-e7aa8d36c574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 18:01:59.945268 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:59.945225    5315 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 18:01:59.945579 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:01:59.945549    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74" podUID="4eeee197-126d-4005-9745-e7aa8d36c574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 18:02:00.948998 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:00.948945    5315 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 18:02:00.949663 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:00.949589    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74" podUID="4eeee197-126d-4005-9745-e7aa8d36c574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 18:02:01.152545 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:01.152506    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c"]
Nov 03 18:02:01.156992 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:01.156960    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c"
Nov 03 18:02:01.217530 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:01.217435    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ac59539-8ac1-466e-8d1b-b2be4c42c708-home\") pod \"distinct-w4-a86228-kserve-58d9bf9c4-44l8c\" (UID: \"8ac59539-8ac1-466e-8d1b-b2be4c42c708\") " pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c"
Nov 03 18:02:01.217530 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:01.217501    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ac59539-8ac1-466e-8d1b-b2be4c42c708-tmp-dir\") pod \"distinct-w4-a86228-kserve-58d9bf9c4-44l8c\" (UID: \"8ac59539-8ac1-466e-8d1b-b2be4c42c708\") " pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c"
Nov 03 18:02:01.217790 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:01.217675    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ac59539-8ac1-466e-8d1b-b2be4c42c708-model-cache\") pod \"distinct-w4-a86228-kserve-58d9bf9c4-44l8c\" (UID: \"8ac59539-8ac1-466e-8d1b-b2be4c42c708\") " pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c"
Nov 03 18:02:01.217790 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:01.217765    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ac59539-8ac1-466e-8d1b-b2be4c42c708-dshm\") pod \"distinct-w4-a86228-kserve-58d9bf9c4-44l8c\" (UID: \"8ac59539-8ac1-466e-8d1b-b2be4c42c708\") " pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c"
Nov 03 18:02:01.217908 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:01.217819    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbh7r\" (UniqueName: \"kubernetes.io/projected/8ac59539-8ac1-466e-8d1b-b2be4c42c708-kube-api-access-hbh7r\") pod \"distinct-w4-a86228-kserve-58d9bf9c4-44l8c\" (UID: \"8ac59539-8ac1-466e-8d1b-b2be4c42c708\") " pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c"
Nov 03 18:02:01.217908 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:01.217885    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ac59539-8ac1-466e-8d1b-b2be4c42c708-tls-certs\") pod \"distinct-w4-a86228-kserve-58d9bf9c4-44l8c\" (UID: \"8ac59539-8ac1-466e-8d1b-b2be4c42c708\") " pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c"
Nov 03 18:02:01.253102 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:01.253067    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-a86228\"/\"distinct-w4-a86228-kserve-self-signed-certs\""
Nov 03 18:02:01.256914 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:01.256878    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c"]
Nov 03 18:02:01.322959 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:01.322913    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ac59539-8ac1-466e-8d1b-b2be4c42c708-dshm\") pod \"distinct-w4-a86228-kserve-58d9bf9c4-44l8c\" (UID: \"8ac59539-8ac1-466e-8d1b-b2be4c42c708\") " pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c"
Nov 03 18:02:01.323233 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:01.323213    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbh7r\" (UniqueName: \"kubernetes.io/projected/8ac59539-8ac1-466e-8d1b-b2be4c42c708-kube-api-access-hbh7r\") pod \"distinct-w4-a86228-kserve-58d9bf9c4-44l8c\" (UID: \"8ac59539-8ac1-466e-8d1b-b2be4c42c708\") " pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c"
Nov 03 18:02:01.323367 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:01.323353    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ac59539-8ac1-466e-8d1b-b2be4c42c708-tls-certs\") pod \"distinct-w4-a86228-kserve-58d9bf9c4-44l8c\" (UID: \"8ac59539-8ac1-466e-8d1b-b2be4c42c708\") " pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c"
Nov 03 18:02:01.323499 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:01.323487    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ac59539-8ac1-466e-8d1b-b2be4c42c708-home\") pod \"distinct-w4-a86228-kserve-58d9bf9c4-44l8c\" (UID: \"8ac59539-8ac1-466e-8d1b-b2be4c42c708\") " pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c"
Nov 03 18:02:01.338133 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:01.323652    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ac59539-8ac1-466e-8d1b-b2be4c42c708-tmp-dir\") pod \"distinct-w4-a86228-kserve-58d9bf9c4-44l8c\" (UID: \"8ac59539-8ac1-466e-8d1b-b2be4c42c708\") " pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c"
Nov 03 18:02:01.338498 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:01.338470    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ac59539-8ac1-466e-8d1b-b2be4c42c708-model-cache\") pod \"distinct-w4-a86228-kserve-58d9bf9c4-44l8c\" (UID: \"8ac59539-8ac1-466e-8d1b-b2be4c42c708\") " pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c"
Nov 03 18:02:01.338650 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:01.336876    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ac59539-8ac1-466e-8d1b-b2be4c42c708-home\") pod \"distinct-w4-a86228-kserve-58d9bf9c4-44l8c\" (UID: \"8ac59539-8ac1-466e-8d1b-b2be4c42c708\") " pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c"
Nov 03 18:02:01.338821 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:01.337995    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ac59539-8ac1-466e-8d1b-b2be4c42c708-tmp-dir\") pod \"distinct-w4-a86228-kserve-58d9bf9c4-44l8c\" (UID: \"8ac59539-8ac1-466e-8d1b-b2be4c42c708\") " pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c"
Nov 03 18:02:01.339187 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:01.339162    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ac59539-8ac1-466e-8d1b-b2be4c42c708-model-cache\") pod \"distinct-w4-a86228-kserve-58d9bf9c4-44l8c\" (UID: \"8ac59539-8ac1-466e-8d1b-b2be4c42c708\") " pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c"
Nov 03 18:02:01.368701 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:01.368657    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ac59539-8ac1-466e-8d1b-b2be4c42c708-tls-certs\") pod \"distinct-w4-a86228-kserve-58d9bf9c4-44l8c\" (UID: \"8ac59539-8ac1-466e-8d1b-b2be4c42c708\") " pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c"
Nov 03 18:02:01.369962 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:01.369923    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ac59539-8ac1-466e-8d1b-b2be4c42c708-dshm\") pod \"distinct-w4-a86228-kserve-58d9bf9c4-44l8c\" (UID: \"8ac59539-8ac1-466e-8d1b-b2be4c42c708\") " pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c"
Nov 03 18:02:01.398056 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:01.398019    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbh7r\" (UniqueName: \"kubernetes.io/projected/8ac59539-8ac1-466e-8d1b-b2be4c42c708-kube-api-access-hbh7r\") pod \"distinct-w4-a86228-kserve-58d9bf9c4-44l8c\" (UID: \"8ac59539-8ac1-466e-8d1b-b2be4c42c708\") " pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c"
Nov 03 18:02:01.470315 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:01.470219    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c"
Nov 03 18:02:01.877248 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:01.877210    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c"]
Nov 03 18:02:01.883227 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:02:01.883189    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ac59539_8ac1_466e_8d1b_b2be4c42c708.slice/crio-cdb341ab0a193afbf77aaafd309c6658b1495a0a14cfbdf70ed15a15fe54dd9c WatchSource:0}: Error finding container cdb341ab0a193afbf77aaafd309c6658b1495a0a14cfbdf70ed15a15fe54dd9c: Status 404 returned error can't find the container with id cdb341ab0a193afbf77aaafd309c6658b1495a0a14cfbdf70ed15a15fe54dd9c
Nov 03 18:02:01.906763 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:01.906730    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c" event={"ID":"8ac59539-8ac1-466e-8d1b-b2be4c42c708","Type":"ContainerStarted","Data":"cdb341ab0a193afbf77aaafd309c6658b1495a0a14cfbdf70ed15a15fe54dd9c"}
Nov 03 18:02:01.946031 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:01.945722    5315 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 18:02:01.946031 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:01.945784    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74" podUID="4eeee197-126d-4005-9745-e7aa8d36c574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 18:02:02.918663 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:02.918622    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c" event={"ID":"8ac59539-8ac1-466e-8d1b-b2be4c42c708","Type":"ContainerStarted","Data":"b8842e0c1bf5eac4427d18af275629e52ff1535fea3e0723abc822cd02f10d77"}
Nov 03 18:02:02.939731 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:02.939695    5315 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 18:02:02.939893 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:02.939772    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74" podUID="4eeee197-126d-4005-9745-e7aa8d36c574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 18:02:03.032059 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:03.032002    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c" podStartSLOduration=2.03198751 podStartE2EDuration="2.03198751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:02:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:02:03.031080505 +0000 UTC m=+962.047278574" watchObservedRunningTime="2026-11-03 18:02:03.03198751 +0000 UTC m=+962.048185616"
Nov 03 18:02:03.470916 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:03.470877    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c"
Nov 03 18:02:03.945041 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:03.945008    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74"
Nov 03 18:02:03.945464 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:03.945344    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74"
Nov 03 18:02:03.946083 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:03.946063    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74"
Nov 03 18:02:11.470744 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:11.470697    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c"
Nov 03 18:02:11.483940 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:11.483907    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c"
Nov 03 18:02:11.990237 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:11.990209    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c"
Nov 03 18:02:24.577780 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:02:24.577738    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:02:38.597573 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:38.597531    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt"]
Nov 03 18:02:38.601222 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:38.601197    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt"
Nov 03 18:02:38.760460 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:38.758477    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-f79c02\"/\"distinct-w2-f79c02-kserve-self-signed-certs\""
Nov 03 18:02:38.763378 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:38.763336    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz8tb\" (UniqueName: \"kubernetes.io/projected/7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625-kube-api-access-wz8tb\") pod \"distinct-w2-f79c02-kserve-6c4b58696b-wxnwt\" (UID: \"7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625\") " pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt"
Nov 03 18:02:38.763549 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:38.763404    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625-dshm\") pod \"distinct-w2-f79c02-kserve-6c4b58696b-wxnwt\" (UID: \"7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625\") " pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt"
Nov 03 18:02:38.763549 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:38.763449    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625-model-cache\") pod \"distinct-w2-f79c02-kserve-6c4b58696b-wxnwt\" (UID: \"7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625\") " pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt"
Nov 03 18:02:38.763549 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:38.763490    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625-tls-certs\") pod \"distinct-w2-f79c02-kserve-6c4b58696b-wxnwt\" (UID: \"7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625\") " pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt"
Nov 03 18:02:38.763747 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:38.763628    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625-home\") pod \"distinct-w2-f79c02-kserve-6c4b58696b-wxnwt\" (UID: \"7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625\") " pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt"
Nov 03 18:02:38.763747 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:38.763677    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625-tmp-dir\") pod \"distinct-w2-f79c02-kserve-6c4b58696b-wxnwt\" (UID: \"7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625\") " pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt"
Nov 03 18:02:38.764381 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:38.764350    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt"]
Nov 03 18:02:38.865244 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:38.865156    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz8tb\" (UniqueName: \"kubernetes.io/projected/7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625-kube-api-access-wz8tb\") pod \"distinct-w2-f79c02-kserve-6c4b58696b-wxnwt\" (UID: \"7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625\") " pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt"
Nov 03 18:02:38.865244 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:38.865212    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625-dshm\") pod \"distinct-w2-f79c02-kserve-6c4b58696b-wxnwt\" (UID: \"7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625\") " pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt"
Nov 03 18:02:38.865543 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:38.865266    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625-model-cache\") pod \"distinct-w2-f79c02-kserve-6c4b58696b-wxnwt\" (UID: \"7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625\") " pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt"
Nov 03 18:02:38.865543 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:38.865376    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625-tls-certs\") pod \"distinct-w2-f79c02-kserve-6c4b58696b-wxnwt\" (UID: \"7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625\") " pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt"
Nov 03 18:02:38.865543 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:38.865517    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625-home\") pod \"distinct-w2-f79c02-kserve-6c4b58696b-wxnwt\" (UID: \"7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625\") " pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt"
Nov 03 18:02:38.865740 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:38.865569    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625-tmp-dir\") pod \"distinct-w2-f79c02-kserve-6c4b58696b-wxnwt\" (UID: \"7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625\") " pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt"
Nov 03 18:02:38.865801 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:38.865742    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625-model-cache\") pod \"distinct-w2-f79c02-kserve-6c4b58696b-wxnwt\" (UID: \"7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625\") " pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt"
Nov 03 18:02:38.865909 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:38.865888    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625-home\") pod \"distinct-w2-f79c02-kserve-6c4b58696b-wxnwt\" (UID: \"7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625\") " pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt"
Nov 03 18:02:38.866027 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:38.866004    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625-tmp-dir\") pod \"distinct-w2-f79c02-kserve-6c4b58696b-wxnwt\" (UID: \"7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625\") " pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt"
Nov 03 18:02:38.868241 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:38.868220    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625-dshm\") pod \"distinct-w2-f79c02-kserve-6c4b58696b-wxnwt\" (UID: \"7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625\") " pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt"
Nov 03 18:02:38.868432 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:38.868413    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625-tls-certs\") pod \"distinct-w2-f79c02-kserve-6c4b58696b-wxnwt\" (UID: \"7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625\") " pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt"
Nov 03 18:02:38.939631 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:38.932584    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz8tb\" (UniqueName: \"kubernetes.io/projected/7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625-kube-api-access-wz8tb\") pod \"distinct-w2-f79c02-kserve-6c4b58696b-wxnwt\" (UID: \"7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625\") " pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt"
Nov 03 18:02:38.945596 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:38.945293    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt"
Nov 03 18:02:39.380353 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:39.380309    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt"]
Nov 03 18:02:39.384458 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:02:39.384402    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d3e8590_15f3_4ca2_ba34_9a8a9d0a9625.slice/crio-356b4b70449b256b9f31b1dbb392c9483d9c1cc3ac777f6cc7babd37233c6b86 WatchSource:0}: Error finding container 356b4b70449b256b9f31b1dbb392c9483d9c1cc3ac777f6cc7babd37233c6b86: Status 404 returned error can't find the container with id 356b4b70449b256b9f31b1dbb392c9483d9c1cc3ac777f6cc7babd37233c6b86
Nov 03 18:02:40.167525 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:40.167484    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt" event={"ID":"7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625","Type":"ContainerStarted","Data":"6b00778efce404a2ba5f3e060b5f1ecab59600daaf00de64fcc29c3c6136a552"}
Nov 03 18:02:40.168087 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:40.167534    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt" event={"ID":"7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625","Type":"ContainerStarted","Data":"356b4b70449b256b9f31b1dbb392c9483d9c1cc3ac777f6cc7babd37233c6b86"}
Nov 03 18:02:40.280822 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:40.280767    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt" podStartSLOduration=2.280752093 podStartE2EDuration="2.280752093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:02:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:02:40.278206944 +0000 UTC m=+999.294405014" watchObservedRunningTime="2026-11-03 18:02:40.280752093 +0000 UTC m=+999.296950207"
Nov 03 18:02:40.946030 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:40.945994    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt"
Nov 03 18:02:48.945951 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:48.945854    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt"
Nov 03 18:02:48.958945 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:48.958917    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt"
Nov 03 18:02:49.256461 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:49.256378    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt"
Nov 03 18:02:54.578005 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:02:54.577960    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:02:57.556891 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:57.556859    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7"]
Nov 03 18:02:57.560044 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:57.560021    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7"
Nov 03 18:02:57.608955 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:57.608913    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-0adcbc\"/\"openshift-service-ca.crt\""
Nov 03 18:02:57.654792 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:57.654751    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-0adcbc\"/\"kube-root-ca.crt\""
Nov 03 18:02:57.654792 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:57.654781    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-0adcbc\"/\"default-dockercfg-tcxw9\""
Nov 03 18:02:57.655051 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:57.654909    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-0adcbc\"/\"distinct-w3-0adcbc-kserve-self-signed-certs\""
Nov 03 18:02:57.658020 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:57.657999    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7"]
Nov 03 18:02:57.675374 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:57.675350    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb-dshm\") pod \"distinct-w3-0adcbc-kserve-7bff5644f5-c55z7\" (UID: \"09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb\") " pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7"
Nov 03 18:02:57.675463 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:57.675383    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb-tmp-dir\") pod \"distinct-w3-0adcbc-kserve-7bff5644f5-c55z7\" (UID: \"09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb\") " pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7"
Nov 03 18:02:57.675512 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:57.675499    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb-tls-certs\") pod \"distinct-w3-0adcbc-kserve-7bff5644f5-c55z7\" (UID: \"09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb\") " pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7"
Nov 03 18:02:57.675551 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:57.675531    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb-model-cache\") pod \"distinct-w3-0adcbc-kserve-7bff5644f5-c55z7\" (UID: \"09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb\") " pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7"
Nov 03 18:02:57.675601 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:57.675550    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmbzb\" (UniqueName: \"kubernetes.io/projected/09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb-kube-api-access-vmbzb\") pod \"distinct-w3-0adcbc-kserve-7bff5644f5-c55z7\" (UID: \"09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb\") " pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7"
Nov 03 18:02:57.675722 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:57.675699    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb-home\") pod \"distinct-w3-0adcbc-kserve-7bff5644f5-c55z7\" (UID: \"09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb\") " pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7"
Nov 03 18:02:57.776706 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:57.776675    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb-tls-certs\") pod \"distinct-w3-0adcbc-kserve-7bff5644f5-c55z7\" (UID: \"09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb\") " pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7"
Nov 03 18:02:57.776891 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:57.776733    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb-model-cache\") pod \"distinct-w3-0adcbc-kserve-7bff5644f5-c55z7\" (UID: \"09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb\") " pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7"
Nov 03 18:02:57.776891 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:57.776770    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmbzb\" (UniqueName: \"kubernetes.io/projected/09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb-kube-api-access-vmbzb\") pod \"distinct-w3-0adcbc-kserve-7bff5644f5-c55z7\" (UID: \"09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb\") " pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7"
Nov 03 18:02:57.776891 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:57.776838    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb-home\") pod \"distinct-w3-0adcbc-kserve-7bff5644f5-c55z7\" (UID: \"09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb\") " pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7"
Nov 03 18:02:57.777062 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:57.776894    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb-dshm\") pod \"distinct-w3-0adcbc-kserve-7bff5644f5-c55z7\" (UID: \"09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb\") " pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7"
Nov 03 18:02:57.777062 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:57.776926    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb-tmp-dir\") pod \"distinct-w3-0adcbc-kserve-7bff5644f5-c55z7\" (UID: \"09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb\") " pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7"
Nov 03 18:02:57.777180 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:57.777166    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb-model-cache\") pod \"distinct-w3-0adcbc-kserve-7bff5644f5-c55z7\" (UID: \"09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb\") " pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7"
Nov 03 18:02:57.777262 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:57.777235    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb-home\") pod \"distinct-w3-0adcbc-kserve-7bff5644f5-c55z7\" (UID: \"09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb\") " pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7"
Nov 03 18:02:57.777350 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:57.777329    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb-tmp-dir\") pod \"distinct-w3-0adcbc-kserve-7bff5644f5-c55z7\" (UID: \"09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb\") " pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7"
Nov 03 18:02:57.779041 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:57.779016    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb-dshm\") pod \"distinct-w3-0adcbc-kserve-7bff5644f5-c55z7\" (UID: \"09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb\") " pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7"
Nov 03 18:02:57.779285 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:57.779268    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb-tls-certs\") pod \"distinct-w3-0adcbc-kserve-7bff5644f5-c55z7\" (UID: \"09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb\") " pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7"
Nov 03 18:02:57.834765 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:57.834687    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmbzb\" (UniqueName: \"kubernetes.io/projected/09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb-kube-api-access-vmbzb\") pod \"distinct-w3-0adcbc-kserve-7bff5644f5-c55z7\" (UID: \"09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb\") " pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7"
Nov 03 18:02:57.871830 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:57.871796    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7"
Nov 03 18:02:58.279853 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:58.279820    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7"]
Nov 03 18:02:58.285505 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:02:58.285473    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09cc9cc8_2f5e_49e8_8134_1e9deec0d6fb.slice/crio-c285ef0b545c5b32b6bd61ee68a761289768d4b04b4b2fd19cf24e9eeb63df3a WatchSource:0}: Error finding container c285ef0b545c5b32b6bd61ee68a761289768d4b04b4b2fd19cf24e9eeb63df3a: Status 404 returned error can't find the container with id c285ef0b545c5b32b6bd61ee68a761289768d4b04b4b2fd19cf24e9eeb63df3a
Nov 03 18:02:58.307928 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:58.307897    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7" event={"ID":"09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb","Type":"ContainerStarted","Data":"c285ef0b545c5b32b6bd61ee68a761289768d4b04b4b2fd19cf24e9eeb63df3a"}
Nov 03 18:02:59.315978 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:59.315944    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7" event={"ID":"09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb","Type":"ContainerStarted","Data":"83357fb07c4a01250285ec6a651a109a9f966ad2f35ef294f0cab0100e8cd9ea"}
Nov 03 18:02:59.427272 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:59.427222    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7" podStartSLOduration=2.427209691 podStartE2EDuration="2.427209691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:02:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:02:59.426260332 +0000 UTC m=+1018.442458401" watchObservedRunningTime="2026-11-03 18:02:59.427209691 +0000 UTC m=+1018.443407759"
Nov 03 18:02:59.872236 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:02:59.872200    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7"
Nov 03 18:03:07.871968 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:03:07.871927    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7"
Nov 03 18:03:07.884842 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:03:07.884812    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7"
Nov 03 18:03:08.387160 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:03:08.387127    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7"
Nov 03 18:03:12.550856 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:03:12.550820    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr"]
Nov 03 18:03:12.554797 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:03:12.554773    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr"
Nov 03 18:03:12.649205 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:03:12.649163    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-2b4f46\"/\"distinct-2-w5-2b4f46-kserve-self-signed-certs\""
Nov 03 18:03:12.651968 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:03:12.651931    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgp2l\" (UniqueName: \"kubernetes.io/projected/08159a02-fc2e-4138-824d-be5ba3e141d8-kube-api-access-vgp2l\") pod \"distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr\" (UID: \"08159a02-fc2e-4138-824d-be5ba3e141d8\") " pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr"
Nov 03 18:03:12.652131 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:03:12.652067    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08159a02-fc2e-4138-824d-be5ba3e141d8-model-cache\") pod \"distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr\" (UID: \"08159a02-fc2e-4138-824d-be5ba3e141d8\") " pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr"
Nov 03 18:03:12.652131 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:03:12.652110    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08159a02-fc2e-4138-824d-be5ba3e141d8-tmp-dir\") pod \"distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr\" (UID: \"08159a02-fc2e-4138-824d-be5ba3e141d8\") " pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr"
Nov 03 18:03:12.652246 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:03:12.652162    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08159a02-fc2e-4138-824d-be5ba3e141d8-tls-certs\") pod \"distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr\" (UID: \"08159a02-fc2e-4138-824d-be5ba3e141d8\") " pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr"
Nov 03 18:03:12.652306 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:03:12.652251    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08159a02-fc2e-4138-824d-be5ba3e141d8-dshm\") pod \"distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr\" (UID: \"08159a02-fc2e-4138-824d-be5ba3e141d8\") " pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr"
Nov 03 18:03:12.652306 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:03:12.652277    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08159a02-fc2e-4138-824d-be5ba3e141d8-home\") pod \"distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr\" (UID: \"08159a02-fc2e-4138-824d-be5ba3e141d8\") " pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr"
Nov 03 18:03:12.652689 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:03:12.652664    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr"]
Nov 03 18:03:12.753278 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:03:12.753238    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgp2l\" (UniqueName: \"kubernetes.io/projected/08159a02-fc2e-4138-824d-be5ba3e141d8-kube-api-access-vgp2l\") pod \"distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr\" (UID: \"08159a02-fc2e-4138-824d-be5ba3e141d8\") " pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr"
Nov 03 18:03:12.753477 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:03:12.753302    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08159a02-fc2e-4138-824d-be5ba3e141d8-model-cache\") pod \"distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr\" (UID: \"08159a02-fc2e-4138-824d-be5ba3e141d8\") " pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr"
Nov 03 18:03:12.753477 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:03:12.753328    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08159a02-fc2e-4138-824d-be5ba3e141d8-tmp-dir\") pod \"distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr\" (UID: \"08159a02-fc2e-4138-824d-be5ba3e141d8\") " pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr"
Nov 03 18:03:12.753477 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:03:12.753364    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08159a02-fc2e-4138-824d-be5ba3e141d8-tls-certs\") pod \"distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr\" (UID: \"08159a02-fc2e-4138-824d-be5ba3e141d8\") " pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr"
Nov 03 18:03:12.753477 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:03:12.753391    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08159a02-fc2e-4138-824d-be5ba3e141d8-dshm\") pod \"distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr\" (UID: \"08159a02-fc2e-4138-824d-be5ba3e141d8\") " pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr"
Nov 03 18:03:12.753477 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:03:12.753421    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08159a02-fc2e-4138-824d-be5ba3e141d8-home\") pod \"distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr\" (UID: \"08159a02-fc2e-4138-824d-be5ba3e141d8\") " pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr"
Nov 03 18:03:12.753800 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:03:12.753710    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08159a02-fc2e-4138-824d-be5ba3e141d8-model-cache\") pod \"distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr\" (UID: \"08159a02-fc2e-4138-824d-be5ba3e141d8\") " pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr"
Nov 03 18:03:12.753895 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:03:12.753875    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08159a02-fc2e-4138-824d-be5ba3e141d8-tmp-dir\") pod \"distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr\" (UID: \"08159a02-fc2e-4138-824d-be5ba3e141d8\") " pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr"
Nov 03 18:03:12.753953 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:03:12.753892    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08159a02-fc2e-4138-824d-be5ba3e141d8-home\") pod \"distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr\" (UID: \"08159a02-fc2e-4138-824d-be5ba3e141d8\") " pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr"
Nov 03 18:03:12.755730 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:03:12.755707    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08159a02-fc2e-4138-824d-be5ba3e141d8-dshm\") pod \"distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr\" (UID: \"08159a02-fc2e-4138-824d-be5ba3e141d8\") " pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr"
Nov 03 18:03:12.756062 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:03:12.755953    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08159a02-fc2e-4138-824d-be5ba3e141d8-tls-certs\") pod \"distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr\" (UID: \"08159a02-fc2e-4138-824d-be5ba3e141d8\") " pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr"
Nov 03 18:03:12.810276 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:03:12.810200    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgp2l\" (UniqueName: \"kubernetes.io/projected/08159a02-fc2e-4138-824d-be5ba3e141d8-kube-api-access-vgp2l\") pod \"distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr\" (UID: \"08159a02-fc2e-4138-824d-be5ba3e141d8\") " pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr"
Nov 03 18:03:12.866211 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:03:12.866176    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr"
Nov 03 18:03:13.298473 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:03:13.298439    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr"]
Nov 03 18:03:13.303492 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:03:13.303460    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08159a02_fc2e_4138_824d_be5ba3e141d8.slice/crio-8e264a1b677f12a4408792bdae6abfa88226dcabfc21c9e1d63cd4441860665b WatchSource:0}: Error finding container 8e264a1b677f12a4408792bdae6abfa88226dcabfc21c9e1d63cd4441860665b: Status 404 returned error can't find the container with id 8e264a1b677f12a4408792bdae6abfa88226dcabfc21c9e1d63cd4441860665b
Nov 03 18:03:13.413144 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:03:13.413107    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr" event={"ID":"08159a02-fc2e-4138-824d-be5ba3e141d8","Type":"ContainerStarted","Data":"23516f2c9599465b328d24128c5d8be7ac1a52e8eee5ef8bdecada9ebcd11397"}
Nov 03 18:03:13.413144 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:03:13.413149    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr" event={"ID":"08159a02-fc2e-4138-824d-be5ba3e141d8","Type":"ContainerStarted","Data":"8e264a1b677f12a4408792bdae6abfa88226dcabfc21c9e1d63cd4441860665b"}
Nov 03 18:03:13.524290 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:03:13.524233    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr" podStartSLOduration=1.5242200320000001 podStartE2EDuration="1.524220032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:03:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:03:13.522093032 +0000 UTC m=+1032.538291101" watchObservedRunningTime="2026-11-03 18:03:13.524220032 +0000 UTC m=+1032.540418100"
Nov 03 18:03:13.887742 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:03:13.887701    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr"
Nov 03 18:03:22.866562 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:03:22.866525    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr"
Nov 03 18:03:22.879976 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:03:22.879942    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr"
Nov 03 18:03:23.488877 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:03:23.488846    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr"
Nov 03 18:03:24.577577 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:03:24.577533    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:03:54.576432 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:03:54.576397    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:04:24.577820 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:04:24.577783    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:04:27.980011 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:27.979974    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff"]
Nov 03 18:04:27.984592 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:27.984568    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff"
Nov 03 18:04:28.076858 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:28.076821    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad113518-40b2-4d26-b886-a7f99afdf6f8-credential-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff\" (UID: \"ad113518-40b2-4d26-b886-a7f99afdf6f8\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff"
Nov 03 18:04:28.077042 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:28.076878    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad113518-40b2-4d26-b886-a7f99afdf6f8-istio-envoy\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff\" (UID: \"ad113518-40b2-4d26-b886-a7f99afdf6f8\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff"
Nov 03 18:04:28.077042 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:28.076957    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z8ld\" (UniqueName: \"kubernetes.io/projected/ad113518-40b2-4d26-b886-a7f99afdf6f8-kube-api-access-4z8ld\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff\" (UID: \"ad113518-40b2-4d26-b886-a7f99afdf6f8\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff"
Nov 03 18:04:28.077042 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:28.077005    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad113518-40b2-4d26-b886-a7f99afdf6f8-workload-certs\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff\" (UID: \"ad113518-40b2-4d26-b886-a7f99afdf6f8\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff"
Nov 03 18:04:28.077188 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:28.077099    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad113518-40b2-4d26-b886-a7f99afdf6f8-istio-token\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff\" (UID: \"ad113518-40b2-4d26-b886-a7f99afdf6f8\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff"
Nov 03 18:04:28.077188 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:28.077161    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad113518-40b2-4d26-b886-a7f99afdf6f8-istio-podinfo\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff\" (UID: \"ad113518-40b2-4d26-b886-a7f99afdf6f8\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff"
Nov 03 18:04:28.077279 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:28.077246    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad113518-40b2-4d26-b886-a7f99afdf6f8-workload-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff\" (UID: \"ad113518-40b2-4d26-b886-a7f99afdf6f8\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff"
Nov 03 18:04:28.077279 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:28.077263    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad113518-40b2-4d26-b886-a7f99afdf6f8-istiod-ca-cert\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff\" (UID: \"ad113518-40b2-4d26-b886-a7f99afdf6f8\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff"
Nov 03 18:04:28.077360 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:28.077295    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad113518-40b2-4d26-b886-a7f99afdf6f8-istio-data\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff\" (UID: \"ad113518-40b2-4d26-b886-a7f99afdf6f8\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff"
Nov 03 18:04:28.083307 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:28.083277    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff"]
Nov 03 18:04:28.179056 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:28.179013    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad113518-40b2-4d26-b886-a7f99afdf6f8-istio-podinfo\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff\" (UID: \"ad113518-40b2-4d26-b886-a7f99afdf6f8\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff"
Nov 03 18:04:28.179316 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:28.179297    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad113518-40b2-4d26-b886-a7f99afdf6f8-workload-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff\" (UID: \"ad113518-40b2-4d26-b886-a7f99afdf6f8\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff"
Nov 03 18:04:28.179423 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:28.179410    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad113518-40b2-4d26-b886-a7f99afdf6f8-istiod-ca-cert\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff\" (UID: \"ad113518-40b2-4d26-b886-a7f99afdf6f8\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff"
Nov 03 18:04:28.179550 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:28.179537    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad113518-40b2-4d26-b886-a7f99afdf6f8-istio-data\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff\" (UID: \"ad113518-40b2-4d26-b886-a7f99afdf6f8\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff"
Nov 03 18:04:28.179701 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:28.179686    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad113518-40b2-4d26-b886-a7f99afdf6f8-credential-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff\" (UID: \"ad113518-40b2-4d26-b886-a7f99afdf6f8\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff"
Nov 03 18:04:28.183634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:28.179799    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad113518-40b2-4d26-b886-a7f99afdf6f8-istio-envoy\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff\" (UID: \"ad113518-40b2-4d26-b886-a7f99afdf6f8\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff"
Nov 03 18:04:28.183868 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:28.183841    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z8ld\" (UniqueName: \"kubernetes.io/projected/ad113518-40b2-4d26-b886-a7f99afdf6f8-kube-api-access-4z8ld\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff\" (UID: \"ad113518-40b2-4d26-b886-a7f99afdf6f8\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff"
Nov 03 18:04:28.206668 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:28.191830    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad113518-40b2-4d26-b886-a7f99afdf6f8-workload-certs\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff\" (UID: \"ad113518-40b2-4d26-b886-a7f99afdf6f8\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff"
Nov 03 18:04:28.206668 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:28.191968    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad113518-40b2-4d26-b886-a7f99afdf6f8-istio-token\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff\" (UID: \"ad113518-40b2-4d26-b886-a7f99afdf6f8\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff"
Nov 03 18:04:28.206668 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:28.187420    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad113518-40b2-4d26-b886-a7f99afdf6f8-istiod-ca-cert\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff\" (UID: \"ad113518-40b2-4d26-b886-a7f99afdf6f8\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff"
Nov 03 18:04:28.206668 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:28.189183    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad113518-40b2-4d26-b886-a7f99afdf6f8-istio-data\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff\" (UID: \"ad113518-40b2-4d26-b886-a7f99afdf6f8\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff"
Nov 03 18:04:28.206668 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:28.192842    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad113518-40b2-4d26-b886-a7f99afdf6f8-workload-certs\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff\" (UID: \"ad113518-40b2-4d26-b886-a7f99afdf6f8\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff"
Nov 03 18:04:28.206668 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:28.188688    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad113518-40b2-4d26-b886-a7f99afdf6f8-workload-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff\" (UID: \"ad113518-40b2-4d26-b886-a7f99afdf6f8\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff"
Nov 03 18:04:28.206668 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:28.188949    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad113518-40b2-4d26-b886-a7f99afdf6f8-credential-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff\" (UID: \"ad113518-40b2-4d26-b886-a7f99afdf6f8\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff"
Nov 03 18:04:28.210633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:28.208231    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad113518-40b2-4d26-b886-a7f99afdf6f8-istio-podinfo\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff\" (UID: \"ad113518-40b2-4d26-b886-a7f99afdf6f8\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff"
Nov 03 18:04:28.211236 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:28.211169    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad113518-40b2-4d26-b886-a7f99afdf6f8-istio-envoy\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff\" (UID: \"ad113518-40b2-4d26-b886-a7f99afdf6f8\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff"
Nov 03 18:04:28.246287 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:28.246211    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z8ld\" (UniqueName: \"kubernetes.io/projected/ad113518-40b2-4d26-b886-a7f99afdf6f8-kube-api-access-4z8ld\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff\" (UID: \"ad113518-40b2-4d26-b886-a7f99afdf6f8\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff"
Nov 03 18:04:28.249485 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:28.249453    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad113518-40b2-4d26-b886-a7f99afdf6f8-istio-token\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff\" (UID: \"ad113518-40b2-4d26-b886-a7f99afdf6f8\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff"
Nov 03 18:04:28.298797 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:28.298764    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff"
Nov 03 18:04:28.724509 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:28.724477    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff"]
Nov 03 18:04:28.731233 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:04:28.731203    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad113518_40b2_4d26_b886_a7f99afdf6f8.slice/crio-f8f0a34dc15635ae895c546915e5fb03efbad3679dce44c7acc3a5b26640b737 WatchSource:0}: Error finding container f8f0a34dc15635ae895c546915e5fb03efbad3679dce44c7acc3a5b26640b737: Status 404 returned error can't find the container with id f8f0a34dc15635ae895c546915e5fb03efbad3679dce44c7acc3a5b26640b737
Nov 03 18:04:28.733729 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:28.733691    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:04:28.733807 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:28.733766    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:04:28.733807 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:28.733798    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:04:28.908161 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:28.908125    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff" event={"ID":"ad113518-40b2-4d26-b886-a7f99afdf6f8","Type":"ContainerStarted","Data":"e6beb19892d462933e2480386ca6c9fb41d3fb186fa0a6e2879c86a73ba7a53d"}
Nov 03 18:04:28.908330 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:28.908184    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff" event={"ID":"ad113518-40b2-4d26-b886-a7f99afdf6f8","Type":"ContainerStarted","Data":"f8f0a34dc15635ae895c546915e5fb03efbad3679dce44c7acc3a5b26640b737"}
Nov 03 18:04:29.022947 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:29.022830    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff" podStartSLOduration=2.022816262 podStartE2EDuration="2.022816262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:04:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:04:29.019732755 +0000 UTC m=+1108.035930824" watchObservedRunningTime="2026-11-03 18:04:29.022816262 +0000 UTC m=+1108.039014330"
Nov 03 18:04:29.299796 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:29.299761    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff"
Nov 03 18:04:29.301407 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:29.301383    5315 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 18:04:29.301473 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:29.301433    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff" podUID="ad113518-40b2-4d26-b886-a7f99afdf6f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 18:04:30.299601 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:30.299562    5315 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 18:04:30.300112 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:30.299659    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff" podUID="ad113518-40b2-4d26-b886-a7f99afdf6f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 18:04:31.299472 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:31.299429    5315 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 18:04:31.299681 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:31.299493    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff" podUID="ad113518-40b2-4d26-b886-a7f99afdf6f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 18:04:31.313974 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:31.313940    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s"]
Nov 03 18:04:31.317879 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:31.317852    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s"
Nov 03 18:04:31.393109 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:31.393075    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-0ccbb9\"/\"kube-root-ca.crt\""
Nov 03 18:04:31.397934 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:31.397909    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-0ccbb9\"/\"distinct-2-w0-0ccbb9-kserve-self-signed-certs\""
Nov 03 18:04:31.413846 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:31.413813    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-0ccbb9\"/\"openshift-service-ca.crt\""
Nov 03 18:04:31.414010 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:31.413856    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-0ccbb9\"/\"default-dockercfg-bw54c\""
Nov 03 18:04:31.417216 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:31.417195    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s"]
Nov 03 18:04:31.452469 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:31.452428    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fnhz\" (UniqueName: \"kubernetes.io/projected/155e4b39-1877-4bd9-8541-5d6bd3aa0a8e-kube-api-access-7fnhz\") pod \"distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s\" (UID: \"155e4b39-1877-4bd9-8541-5d6bd3aa0a8e\") " pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s"
Nov 03 18:04:31.452689 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:31.452560    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/155e4b39-1877-4bd9-8541-5d6bd3aa0a8e-dshm\") pod \"distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s\" (UID: \"155e4b39-1877-4bd9-8541-5d6bd3aa0a8e\") " pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s"
Nov 03 18:04:31.452689 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:31.452665    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/155e4b39-1877-4bd9-8541-5d6bd3aa0a8e-model-cache\") pod \"distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s\" (UID: \"155e4b39-1877-4bd9-8541-5d6bd3aa0a8e\") " pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s"
Nov 03 18:04:31.452790 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:31.452773    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/155e4b39-1877-4bd9-8541-5d6bd3aa0a8e-home\") pod \"distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s\" (UID: \"155e4b39-1877-4bd9-8541-5d6bd3aa0a8e\") " pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s"
Nov 03 18:04:31.452848 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:31.452830    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/155e4b39-1877-4bd9-8541-5d6bd3aa0a8e-tls-certs\") pod \"distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s\" (UID: \"155e4b39-1877-4bd9-8541-5d6bd3aa0a8e\") " pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s"
Nov 03 18:04:31.452903 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:31.452876    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/155e4b39-1877-4bd9-8541-5d6bd3aa0a8e-tmp-dir\") pod \"distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s\" (UID: \"155e4b39-1877-4bd9-8541-5d6bd3aa0a8e\") " pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s"
Nov 03 18:04:31.554361 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:31.554269    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/155e4b39-1877-4bd9-8541-5d6bd3aa0a8e-dshm\") pod \"distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s\" (UID: \"155e4b39-1877-4bd9-8541-5d6bd3aa0a8e\") " pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s"
Nov 03 18:04:31.554361 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:31.554341    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/155e4b39-1877-4bd9-8541-5d6bd3aa0a8e-model-cache\") pod \"distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s\" (UID: \"155e4b39-1877-4bd9-8541-5d6bd3aa0a8e\") " pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s"
Nov 03 18:04:31.554584 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:31.554436    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/155e4b39-1877-4bd9-8541-5d6bd3aa0a8e-home\") pod \"distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s\" (UID: \"155e4b39-1877-4bd9-8541-5d6bd3aa0a8e\") " pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s"
Nov 03 18:04:31.554584 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:31.554475    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/155e4b39-1877-4bd9-8541-5d6bd3aa0a8e-tls-certs\") pod \"distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s\" (UID: \"155e4b39-1877-4bd9-8541-5d6bd3aa0a8e\") " pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s"
Nov 03 18:04:31.554584 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:31.554515    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/155e4b39-1877-4bd9-8541-5d6bd3aa0a8e-tmp-dir\") pod \"distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s\" (UID: \"155e4b39-1877-4bd9-8541-5d6bd3aa0a8e\") " pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s"
Nov 03 18:04:31.554777 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:31.554599    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fnhz\" (UniqueName: \"kubernetes.io/projected/155e4b39-1877-4bd9-8541-5d6bd3aa0a8e-kube-api-access-7fnhz\") pod \"distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s\" (UID: \"155e4b39-1877-4bd9-8541-5d6bd3aa0a8e\") " pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s"
Nov 03 18:04:31.554929 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:31.554907    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/155e4b39-1877-4bd9-8541-5d6bd3aa0a8e-model-cache\") pod \"distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s\" (UID: \"155e4b39-1877-4bd9-8541-5d6bd3aa0a8e\") " pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s"
Nov 03 18:04:31.555085 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:31.555058    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/155e4b39-1877-4bd9-8541-5d6bd3aa0a8e-tmp-dir\") pod \"distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s\" (UID: \"155e4b39-1877-4bd9-8541-5d6bd3aa0a8e\") " pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s"
Nov 03 18:04:31.555176 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:31.555076    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/155e4b39-1877-4bd9-8541-5d6bd3aa0a8e-home\") pod \"distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s\" (UID: \"155e4b39-1877-4bd9-8541-5d6bd3aa0a8e\") " pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s"
Nov 03 18:04:31.556686 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:31.556664    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/155e4b39-1877-4bd9-8541-5d6bd3aa0a8e-dshm\") pod \"distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s\" (UID: \"155e4b39-1877-4bd9-8541-5d6bd3aa0a8e\") " pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s"
Nov 03 18:04:31.557237 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:31.557217    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/155e4b39-1877-4bd9-8541-5d6bd3aa0a8e-tls-certs\") pod \"distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s\" (UID: \"155e4b39-1877-4bd9-8541-5d6bd3aa0a8e\") " pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s"
Nov 03 18:04:31.612061 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:31.612028    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fnhz\" (UniqueName: \"kubernetes.io/projected/155e4b39-1877-4bd9-8541-5d6bd3aa0a8e-kube-api-access-7fnhz\") pod \"distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s\" (UID: \"155e4b39-1877-4bd9-8541-5d6bd3aa0a8e\") " pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s"
Nov 03 18:04:31.654591 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:31.654555    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s"
Nov 03 18:04:32.144994 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:32.144897    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s"]
Nov 03 18:04:32.150770 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:04:32.150730    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod155e4b39_1877_4bd9_8541_5d6bd3aa0a8e.slice/crio-e7563dba898af3db10bb674438756cd7fbaa27b4ac7df591b3b48fa3dffa0526 WatchSource:0}: Error finding container e7563dba898af3db10bb674438756cd7fbaa27b4ac7df591b3b48fa3dffa0526: Status 404 returned error can't find the container with id e7563dba898af3db10bb674438756cd7fbaa27b4ac7df591b3b48fa3dffa0526
Nov 03 18:04:32.299515 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:32.299485    5315 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 18:04:32.299718 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:32.299543    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff" podUID="ad113518-40b2-4d26-b886-a7f99afdf6f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 18:04:32.960436 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:32.960395    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s" event={"ID":"155e4b39-1877-4bd9-8541-5d6bd3aa0a8e","Type":"ContainerStarted","Data":"b45322fa2f2e8a299cd8514f46f06c912e78288d420ef1555b58f0746a8f5813"}
Nov 03 18:04:32.960634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:32.960450    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s" event={"ID":"155e4b39-1877-4bd9-8541-5d6bd3aa0a8e","Type":"ContainerStarted","Data":"e7563dba898af3db10bb674438756cd7fbaa27b4ac7df591b3b48fa3dffa0526"}
Nov 03 18:04:33.071360 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:33.071297    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s" podStartSLOduration=2.071281221 podStartE2EDuration="2.071281221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:04:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:04:33.069524256 +0000 UTC m=+1112.085722326" watchObservedRunningTime="2026-11-03 18:04:33.071281221 +0000 UTC m=+1112.087479290"
Nov 03 18:04:33.299416 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:33.299374    5315 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 18:04:33.299714 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:33.299438    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff" podUID="ad113518-40b2-4d26-b886-a7f99afdf6f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 18:04:33.655275 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:33.655175    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s"
Nov 03 18:04:34.299745 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:34.299706    5315 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 18:04:34.299933 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:34.299767    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff" podUID="ad113518-40b2-4d26-b886-a7f99afdf6f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 18:04:35.300155 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:35.300113    5315 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 18:04:35.300549 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:35.300184    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff" podUID="ad113518-40b2-4d26-b886-a7f99afdf6f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 18:04:36.304010 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:36.303979    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff"
Nov 03 18:04:36.304511 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:36.304397    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff"
Nov 03 18:04:36.305283 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:36.305261    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff"
Nov 03 18:04:41.655803 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:41.655760    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s"
Nov 03 18:04:41.668231 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:41.668208    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s"
Nov 03 18:04:42.033191 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:42.033164    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s"
Nov 03 18:04:48.827894 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:48.827853    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c44d52e-ldxbf"]
Nov 03 18:04:48.832499 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:48.832474    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c44d52e-ldxbf"
Nov 03 18:04:48.880937 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:48.880898    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:04:48.881148 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:48.881059    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:04:48.926970 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:48.926933    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:04:48.926970 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:48.926945    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:04:48.932411 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:48.932382    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c44d52e-ldxbf"]
Nov 03 18:04:48.983882 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:48.981941    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bc89\" (UniqueName: \"kubernetes.io/projected/cb88a0c1-77c2-495a-bf00-168ce82fa90e-kube-api-access-2bc89\") pod \"maas-api-revoke-sub-1c44d52e-ldxbf\" (UID: \"cb88a0c1-77c2-495a-bf00-168ce82fa90e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c44d52e-ldxbf"
Nov 03 18:04:48.983882 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:48.982019    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb88a0c1-77c2-495a-bf00-168ce82fa90e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c44d52e-ldxbf\" (UID: \"cb88a0c1-77c2-495a-bf00-168ce82fa90e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c44d52e-ldxbf"
Nov 03 18:04:48.983882 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:48.982194    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb88a0c1-77c2-495a-bf00-168ce82fa90e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c44d52e-ldxbf\" (UID: \"cb88a0c1-77c2-495a-bf00-168ce82fa90e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c44d52e-ldxbf"
Nov 03 18:04:49.084090 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:49.083997    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bc89\" (UniqueName: \"kubernetes.io/projected/cb88a0c1-77c2-495a-bf00-168ce82fa90e-kube-api-access-2bc89\") pod \"maas-api-revoke-sub-1c44d52e-ldxbf\" (UID: \"cb88a0c1-77c2-495a-bf00-168ce82fa90e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c44d52e-ldxbf"
Nov 03 18:04:49.084090 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:49.084043    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb88a0c1-77c2-495a-bf00-168ce82fa90e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c44d52e-ldxbf\" (UID: \"cb88a0c1-77c2-495a-bf00-168ce82fa90e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c44d52e-ldxbf"
Nov 03 18:04:49.084273 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:49.084177    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb88a0c1-77c2-495a-bf00-168ce82fa90e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c44d52e-ldxbf\" (UID: \"cb88a0c1-77c2-495a-bf00-168ce82fa90e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c44d52e-ldxbf"
Nov 03 18:04:49.084790 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:49.084766    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb88a0c1-77c2-495a-bf00-168ce82fa90e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c44d52e-ldxbf\" (UID: \"cb88a0c1-77c2-495a-bf00-168ce82fa90e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c44d52e-ldxbf"
Nov 03 18:04:49.086458 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:49.086440    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb88a0c1-77c2-495a-bf00-168ce82fa90e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c44d52e-ldxbf\" (UID: \"cb88a0c1-77c2-495a-bf00-168ce82fa90e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c44d52e-ldxbf"
Nov 03 18:04:49.141569 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:49.141527    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bc89\" (UniqueName: \"kubernetes.io/projected/cb88a0c1-77c2-495a-bf00-168ce82fa90e-kube-api-access-2bc89\") pod \"maas-api-revoke-sub-1c44d52e-ldxbf\" (UID: \"cb88a0c1-77c2-495a-bf00-168ce82fa90e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c44d52e-ldxbf"
Nov 03 18:04:49.147231 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:49.147206    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c44d52e-ldxbf"
Nov 03 18:04:49.520197 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:49.520102    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c44d52e-ldxbf"]
Nov 03 18:04:49.528730 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:04:49.528697    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb88a0c1_77c2_495a_bf00_168ce82fa90e.slice/crio-c6fadd29f24c7cd203fcdeb50fa6221d67c25aaa1ab72e05ee13ab8857249f12 WatchSource:0}: Error finding container c6fadd29f24c7cd203fcdeb50fa6221d67c25aaa1ab72e05ee13ab8857249f12: Status 404 returned error can't find the container with id c6fadd29f24c7cd203fcdeb50fa6221d67c25aaa1ab72e05ee13ab8857249f12
Nov 03 18:04:50.063854 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:50.063815    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d6c743e-jfx4h"]
Nov 03 18:04:50.067008 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:50.066981    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d6c743e-jfx4h"]
Nov 03 18:04:50.079567 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:50.079532    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c44d52e-ldxbf" event={"ID":"cb88a0c1-77c2-495a-bf00-168ce82fa90e","Type":"ContainerStarted","Data":"9967728e3521f5dbfc2fe4388675b4dc7fbcea9b33967bcb4cbffa9d71b46d70"}
Nov 03 18:04:50.079778 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:50.079592    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c44d52e-ldxbf" event={"ID":"cb88a0c1-77c2-495a-bf00-168ce82fa90e","Type":"ContainerStarted","Data":"c6fadd29f24c7cd203fcdeb50fa6221d67c25aaa1ab72e05ee13ab8857249f12"}
Nov 03 18:04:50.186997 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:50.186888    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c44d52e-ldxbf" podStartSLOduration=2.186871344 podStartE2EDuration="2.186871344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:04:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:04:50.186392587 +0000 UTC m=+1129.202590656" watchObservedRunningTime="2026-11-03 18:04:50.186871344 +0000 UTC m=+1129.203069413"
Nov 03 18:04:51.089027 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:51.088988    5315 generic.go:356] "Generic (PLEG): container finished" podID="cb88a0c1-77c2-495a-bf00-168ce82fa90e" containerID="9967728e3521f5dbfc2fe4388675b4dc7fbcea9b33967bcb4cbffa9d71b46d70" exitCode=0
Nov 03 18:04:51.089650 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:51.089048    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c44d52e-ldxbf" event={"ID":"cb88a0c1-77c2-495a-bf00-168ce82fa90e","Type":"ContainerDied","Data":"9967728e3521f5dbfc2fe4388675b4dc7fbcea9b33967bcb4cbffa9d71b46d70"}
Nov 03 18:04:51.104991 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:51.104955    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40bf5d8c-f779-48b5-8273-e7ae30a98537" path="/var/lib/kubelet/pods/40bf5d8c-f779-48b5-8273-e7ae30a98537/volumes"
Nov 03 18:04:52.248243 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:52.248208    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c44d52e-ldxbf"
Nov 03 18:04:52.348663 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:52.348554    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb88a0c1-77c2-495a-bf00-168ce82fa90e-kube-api-access-2bc89\" (UniqueName: \"kubernetes.io/projected/cb88a0c1-77c2-495a-bf00-168ce82fa90e-kube-api-access-2bc89\") pod \"cb88a0c1-77c2-495a-bf00-168ce82fa90e\" (UID: \"cb88a0c1-77c2-495a-bf00-168ce82fa90e\") "
Nov 03 18:04:52.348819 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:52.348708    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb88a0c1-77c2-495a-bf00-168ce82fa90e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb88a0c1-77c2-495a-bf00-168ce82fa90e-maas-api-openshift-ca\") pod \"cb88a0c1-77c2-495a-bf00-168ce82fa90e\" (UID: \"cb88a0c1-77c2-495a-bf00-168ce82fa90e\") "
Nov 03 18:04:52.348819 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:52.348730    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb88a0c1-77c2-495a-bf00-168ce82fa90e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb88a0c1-77c2-495a-bf00-168ce82fa90e-maas-api-xks-ca\") pod \"cb88a0c1-77c2-495a-bf00-168ce82fa90e\" (UID: \"cb88a0c1-77c2-495a-bf00-168ce82fa90e\") "
Nov 03 18:04:52.349044 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:52.349022    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb88a0c1-77c2-495a-bf00-168ce82fa90e-maas-api-openshift-ca" pod "cb88a0c1-77c2-495a-bf00-168ce82fa90e" (UID: "cb88a0c1-77c2-495a-bf00-168ce82fa90e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:04:52.349102 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:52.349084    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb88a0c1-77c2-495a-bf00-168ce82fa90e-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:04:52.350777 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:52.350753    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb88a0c1-77c2-495a-bf00-168ce82fa90e-maas-api-xks-ca" pod "cb88a0c1-77c2-495a-bf00-168ce82fa90e" (UID: "cb88a0c1-77c2-495a-bf00-168ce82fa90e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:04:52.350922 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:52.350900    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb88a0c1-77c2-495a-bf00-168ce82fa90e-kube-api-access-2bc89" pod "cb88a0c1-77c2-495a-bf00-168ce82fa90e" (UID: "cb88a0c1-77c2-495a-bf00-168ce82fa90e"). InnerVolumeSpecName "kube-api-access-2bc89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:04:52.449881 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:52.449844    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bc89\" (UniqueName: \"kubernetes.io/projected/cb88a0c1-77c2-495a-bf00-168ce82fa90e-kube-api-access-2bc89\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:04:52.449881 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:52.449880    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb88a0c1-77c2-495a-bf00-168ce82fa90e-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:04:53.104790 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:53.104758    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c44d52e-ldxbf"
Nov 03 18:04:53.105122 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:53.105097    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c44d52e-ldxbf" event={"ID":"cb88a0c1-77c2-495a-bf00-168ce82fa90e","Type":"ContainerDied","Data":"c6fadd29f24c7cd203fcdeb50fa6221d67c25aaa1ab72e05ee13ab8857249f12"}
Nov 03 18:04:53.105182 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:53.105146    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6fadd29f24c7cd203fcdeb50fa6221d67c25aaa1ab72e05ee13ab8857249f12"
Nov 03 18:04:54.578741 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:04:54.578703    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:04:55.050348 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:55.050309    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85b3e2e2-z982z"]
Nov 03 18:04:55.094626 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:55.094576    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85b3e2e2-z982z"]
Nov 03 18:04:55.103834 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:04:55.103806    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="402fd518-42cc-4242-8e14-96e762de3f13" path="/var/lib/kubelet/pods/402fd518-42cc-4242-8e14-96e762de3f13/volumes"
Nov 03 18:05:01.050722 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:01.050680    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d3ab78f-bvrnk"]
Nov 03 18:05:01.103826 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:01.103797    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d3ab78f-bvrnk"]
Nov 03 18:05:01.147736 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:05:01.147694    5315 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-9d3ab78f-bvrnk\" is forbidden: User \"system:node:ip-10-0-128-204.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-128-204.us-west-2.compute.internal' and this object" podUID="cf859d09-eac4-4450-9269-e8cb2c3020f5" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d3ab78f-bvrnk"
Nov 03 18:05:03.109770 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:03.109731    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf859d09-eac4-4450-9269-e8cb2c3020f5" path="/var/lib/kubelet/pods/cf859d09-eac4-4450-9269-e8cb2c3020f5/volumes"
Nov 03 18:05:04.548552 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:04.548513    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96cc1fde-txnlx"]
Nov 03 18:05:04.551297 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:04.550392    5315 cpu_manager.go:414] "containerMap: removing container" podUID="cb88a0c1-77c2-495a-bf00-168ce82fa90e" containerName="revoke-keys"
Nov 03 18:05:04.551297 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:04.550422    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb88a0c1-77c2-495a-bf00-168ce82fa90e" containerName="revoke-keys"
Nov 03 18:05:04.553642 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:04.551588    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb88a0c1-77c2-495a-bf00-168ce82fa90e" containerName="revoke-keys"
Nov 03 18:05:04.557652 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:04.557629    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96cc1fde-txnlx"
Nov 03 18:05:04.608497 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:04.608465    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:05:04.608753 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:04.608735    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:05:04.608821 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:04.608735    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:05:04.608821 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:04.608772    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:05:04.645662 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:04.645624    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d87d88-8b0e-4e14-9fa0-cc704c31e415-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96cc1fde-txnlx\" (UID: \"23d87d88-8b0e-4e14-9fa0-cc704c31e415\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96cc1fde-txnlx"
Nov 03 18:05:04.645831 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:04.645749    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d87d88-8b0e-4e14-9fa0-cc704c31e415-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96cc1fde-txnlx\" (UID: \"23d87d88-8b0e-4e14-9fa0-cc704c31e415\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96cc1fde-txnlx"
Nov 03 18:05:04.645831 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:04.645808    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kt4g\" (UniqueName: \"kubernetes.io/projected/23d87d88-8b0e-4e14-9fa0-cc704c31e415-kube-api-access-7kt4g\") pod \"maas-api-revoke-sub-96cc1fde-txnlx\" (UID: \"23d87d88-8b0e-4e14-9fa0-cc704c31e415\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96cc1fde-txnlx"
Nov 03 18:05:04.657663 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:04.657632    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96cc1fde-txnlx"]
Nov 03 18:05:04.747211 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:04.747175    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d87d88-8b0e-4e14-9fa0-cc704c31e415-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96cc1fde-txnlx\" (UID: \"23d87d88-8b0e-4e14-9fa0-cc704c31e415\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96cc1fde-txnlx"
Nov 03 18:05:04.747419 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:04.747262    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d87d88-8b0e-4e14-9fa0-cc704c31e415-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96cc1fde-txnlx\" (UID: \"23d87d88-8b0e-4e14-9fa0-cc704c31e415\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96cc1fde-txnlx"
Nov 03 18:05:04.747419 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:04.747284    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kt4g\" (UniqueName: \"kubernetes.io/projected/23d87d88-8b0e-4e14-9fa0-cc704c31e415-kube-api-access-7kt4g\") pod \"maas-api-revoke-sub-96cc1fde-txnlx\" (UID: \"23d87d88-8b0e-4e14-9fa0-cc704c31e415\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96cc1fde-txnlx"
Nov 03 18:05:04.747981 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:04.747955    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d87d88-8b0e-4e14-9fa0-cc704c31e415-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96cc1fde-txnlx\" (UID: \"23d87d88-8b0e-4e14-9fa0-cc704c31e415\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96cc1fde-txnlx"
Nov 03 18:05:04.749761 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:04.749739    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d87d88-8b0e-4e14-9fa0-cc704c31e415-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96cc1fde-txnlx\" (UID: \"23d87d88-8b0e-4e14-9fa0-cc704c31e415\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96cc1fde-txnlx"
Nov 03 18:05:04.804873 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:04.804805    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kt4g\" (UniqueName: \"kubernetes.io/projected/23d87d88-8b0e-4e14-9fa0-cc704c31e415-kube-api-access-7kt4g\") pod \"maas-api-revoke-sub-96cc1fde-txnlx\" (UID: \"23d87d88-8b0e-4e14-9fa0-cc704c31e415\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96cc1fde-txnlx"
Nov 03 18:05:04.870896 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:04.870862    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96cc1fde-txnlx"
Nov 03 18:05:05.248243 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:05.248125    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96cc1fde-txnlx"]
Nov 03 18:05:05.253329 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:05:05.253303    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23d87d88_8b0e_4e14_9fa0_cc704c31e415.slice/crio-ed160ca9dafb5789a8c25208e79614e6f05703d1a43d3f3af27ad9d6c76842f5 WatchSource:0}: Error finding container ed160ca9dafb5789a8c25208e79614e6f05703d1a43d3f3af27ad9d6c76842f5: Status 404 returned error can't find the container with id ed160ca9dafb5789a8c25208e79614e6f05703d1a43d3f3af27ad9d6c76842f5
Nov 03 18:05:06.192107 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:06.192068    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96cc1fde-txnlx" event={"ID":"23d87d88-8b0e-4e14-9fa0-cc704c31e415","Type":"ContainerStarted","Data":"0b714255af6ea54188c66fa9f4c2996c5423a28e2c49c30b2e28aff503f5d57e"}
Nov 03 18:05:06.192511 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:06.192131    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96cc1fde-txnlx" event={"ID":"23d87d88-8b0e-4e14-9fa0-cc704c31e415","Type":"ContainerStarted","Data":"ed160ca9dafb5789a8c25208e79614e6f05703d1a43d3f3af27ad9d6c76842f5"}
Nov 03 18:05:06.304195 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:06.304135    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96cc1fde-txnlx" podStartSLOduration=2.304118449 podStartE2EDuration="2.304118449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:05:06.303922932 +0000 UTC m=+1145.320121001" watchObservedRunningTime="2026-11-03 18:05:06.304118449 +0000 UTC m=+1145.320316542"
Nov 03 18:05:07.199629 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:07.199575    5315 generic.go:356] "Generic (PLEG): container finished" podID="23d87d88-8b0e-4e14-9fa0-cc704c31e415" containerID="0b714255af6ea54188c66fa9f4c2996c5423a28e2c49c30b2e28aff503f5d57e" exitCode=0
Nov 03 18:05:07.200034 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:07.199648    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96cc1fde-txnlx" event={"ID":"23d87d88-8b0e-4e14-9fa0-cc704c31e415","Type":"ContainerDied","Data":"0b714255af6ea54188c66fa9f4c2996c5423a28e2c49c30b2e28aff503f5d57e"}
Nov 03 18:05:07.990087 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:07.990048    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5e4da69-lqlb9"]
Nov 03 18:05:07.995674 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:07.995644    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e4da69-lqlb9"
Nov 03 18:05:08.054299 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:08.054270    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5e4da69-lqlb9"]
Nov 03 18:05:08.092692 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:08.092651    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj74h\" (UniqueName: \"kubernetes.io/projected/806db06a-f8d1-43cd-9a9a-452dbdc0c949-kube-api-access-jj74h\") pod \"maas-api-revoke-sub-f5e4da69-lqlb9\" (UID: \"806db06a-f8d1-43cd-9a9a-452dbdc0c949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e4da69-lqlb9"
Nov 03 18:05:08.092888 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:08.092797    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/806db06a-f8d1-43cd-9a9a-452dbdc0c949-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5e4da69-lqlb9\" (UID: \"806db06a-f8d1-43cd-9a9a-452dbdc0c949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e4da69-lqlb9"
Nov 03 18:05:08.092984 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:08.092962    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/806db06a-f8d1-43cd-9a9a-452dbdc0c949-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5e4da69-lqlb9\" (UID: \"806db06a-f8d1-43cd-9a9a-452dbdc0c949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e4da69-lqlb9"
Nov 03 18:05:08.194484 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:08.194447    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/806db06a-f8d1-43cd-9a9a-452dbdc0c949-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5e4da69-lqlb9\" (UID: \"806db06a-f8d1-43cd-9a9a-452dbdc0c949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e4da69-lqlb9"
Nov 03 18:05:08.194681 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:08.194533    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj74h\" (UniqueName: \"kubernetes.io/projected/806db06a-f8d1-43cd-9a9a-452dbdc0c949-kube-api-access-jj74h\") pod \"maas-api-revoke-sub-f5e4da69-lqlb9\" (UID: \"806db06a-f8d1-43cd-9a9a-452dbdc0c949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e4da69-lqlb9"
Nov 03 18:05:08.194681 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:08.194623    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/806db06a-f8d1-43cd-9a9a-452dbdc0c949-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5e4da69-lqlb9\" (UID: \"806db06a-f8d1-43cd-9a9a-452dbdc0c949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e4da69-lqlb9"
Nov 03 18:05:08.195119 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:08.195095    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/806db06a-f8d1-43cd-9a9a-452dbdc0c949-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5e4da69-lqlb9\" (UID: \"806db06a-f8d1-43cd-9a9a-452dbdc0c949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e4da69-lqlb9"
Nov 03 18:05:08.196887 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:08.196863    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/806db06a-f8d1-43cd-9a9a-452dbdc0c949-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5e4da69-lqlb9\" (UID: \"806db06a-f8d1-43cd-9a9a-452dbdc0c949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e4da69-lqlb9"
Nov 03 18:05:08.255403 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:08.255315    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj74h\" (UniqueName: \"kubernetes.io/projected/806db06a-f8d1-43cd-9a9a-452dbdc0c949-kube-api-access-jj74h\") pod \"maas-api-revoke-sub-f5e4da69-lqlb9\" (UID: \"806db06a-f8d1-43cd-9a9a-452dbdc0c949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e4da69-lqlb9"
Nov 03 18:05:08.309336 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:08.309299    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e4da69-lqlb9"
Nov 03 18:05:08.368155 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:08.365974    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96cc1fde-txnlx"
Nov 03 18:05:08.496931 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:08.496890    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23d87d88-8b0e-4e14-9fa0-cc704c31e415-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d87d88-8b0e-4e14-9fa0-cc704c31e415-maas-api-xks-ca\") pod \"23d87d88-8b0e-4e14-9fa0-cc704c31e415\" (UID: \"23d87d88-8b0e-4e14-9fa0-cc704c31e415\") "
Nov 03 18:05:08.497116 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:08.497037    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23d87d88-8b0e-4e14-9fa0-cc704c31e415-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d87d88-8b0e-4e14-9fa0-cc704c31e415-maas-api-openshift-ca\") pod \"23d87d88-8b0e-4e14-9fa0-cc704c31e415\" (UID: \"23d87d88-8b0e-4e14-9fa0-cc704c31e415\") "
Nov 03 18:05:08.497116 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:08.497107    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23d87d88-8b0e-4e14-9fa0-cc704c31e415-kube-api-access-7kt4g\" (UniqueName: \"kubernetes.io/projected/23d87d88-8b0e-4e14-9fa0-cc704c31e415-kube-api-access-7kt4g\") pod \"23d87d88-8b0e-4e14-9fa0-cc704c31e415\" (UID: \"23d87d88-8b0e-4e14-9fa0-cc704c31e415\") "
Nov 03 18:05:08.497447 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:08.497420    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23d87d88-8b0e-4e14-9fa0-cc704c31e415-maas-api-openshift-ca" pod "23d87d88-8b0e-4e14-9fa0-cc704c31e415" (UID: "23d87d88-8b0e-4e14-9fa0-cc704c31e415"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:05:08.497668 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:08.497642    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d87d88-8b0e-4e14-9fa0-cc704c31e415-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:05:08.499405 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:08.499375    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23d87d88-8b0e-4e14-9fa0-cc704c31e415-kube-api-access-7kt4g" pod "23d87d88-8b0e-4e14-9fa0-cc704c31e415" (UID: "23d87d88-8b0e-4e14-9fa0-cc704c31e415"). InnerVolumeSpecName "kube-api-access-7kt4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:05:08.499527 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:08.499511    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23d87d88-8b0e-4e14-9fa0-cc704c31e415-maas-api-xks-ca" pod "23d87d88-8b0e-4e14-9fa0-cc704c31e415" (UID: "23d87d88-8b0e-4e14-9fa0-cc704c31e415"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:05:08.598527 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:08.598494    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kt4g\" (UniqueName: \"kubernetes.io/projected/23d87d88-8b0e-4e14-9fa0-cc704c31e415-kube-api-access-7kt4g\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:05:08.598527 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:08.598525    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d87d88-8b0e-4e14-9fa0-cc704c31e415-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:05:08.691321 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:08.691284    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5e4da69-lqlb9"]
Nov 03 18:05:08.696855 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:05:08.696822    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod806db06a_f8d1_43cd_9a9a_452dbdc0c949.slice/crio-23382d57a1ab2f1a257b0b7755bfb65ead45038cabc567032f95ecb950b42850 WatchSource:0}: Error finding container 23382d57a1ab2f1a257b0b7755bfb65ead45038cabc567032f95ecb950b42850: Status 404 returned error can't find the container with id 23382d57a1ab2f1a257b0b7755bfb65ead45038cabc567032f95ecb950b42850
Nov 03 18:05:09.213103 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:09.213070    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96cc1fde-txnlx"
Nov 03 18:05:09.213238 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:09.213073    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96cc1fde-txnlx" event={"ID":"23d87d88-8b0e-4e14-9fa0-cc704c31e415","Type":"ContainerDied","Data":"ed160ca9dafb5789a8c25208e79614e6f05703d1a43d3f3af27ad9d6c76842f5"}
Nov 03 18:05:09.213238 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:09.213170    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed160ca9dafb5789a8c25208e79614e6f05703d1a43d3f3af27ad9d6c76842f5"
Nov 03 18:05:09.214717 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:09.214693    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e4da69-lqlb9" event={"ID":"806db06a-f8d1-43cd-9a9a-452dbdc0c949","Type":"ContainerStarted","Data":"a7eded8a39bf6654e1279aff2ed9e2edf2843a17f797958b84e640cbef9b69d9"}
Nov 03 18:05:09.214849 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:09.214732    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e4da69-lqlb9" event={"ID":"806db06a-f8d1-43cd-9a9a-452dbdc0c949","Type":"ContainerStarted","Data":"23382d57a1ab2f1a257b0b7755bfb65ead45038cabc567032f95ecb950b42850"}
Nov 03 18:05:09.329497 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:09.329440    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e4da69-lqlb9" podStartSLOduration=2.329425484 podStartE2EDuration="2.329425484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:05:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:05:09.328817832 +0000 UTC m=+1148.345015900" watchObservedRunningTime="2026-11-03 18:05:09.329425484 +0000 UTC m=+1148.345623596"
Nov 03 18:05:10.223422 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:10.223386    5315 generic.go:356] "Generic (PLEG): container finished" podID="806db06a-f8d1-43cd-9a9a-452dbdc0c949" containerID="a7eded8a39bf6654e1279aff2ed9e2edf2843a17f797958b84e640cbef9b69d9" exitCode=0
Nov 03 18:05:10.223642 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:10.223480    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e4da69-lqlb9" event={"ID":"806db06a-f8d1-43cd-9a9a-452dbdc0c949","Type":"ContainerDied","Data":"a7eded8a39bf6654e1279aff2ed9e2edf2843a17f797958b84e640cbef9b69d9"}
Nov 03 18:05:11.050090 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:11.050047    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6af2fee0-hhwhq"]
Nov 03 18:05:11.095934 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:11.095901    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6af2fee0-hhwhq"]
Nov 03 18:05:11.107837 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:11.107800    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="590eae70-e712-474f-8b0c-dc6cb1873e63" path="/var/lib/kubelet/pods/590eae70-e712-474f-8b0c-dc6cb1873e63/volumes"
Nov 03 18:05:11.379718 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:11.379686    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e4da69-lqlb9"
Nov 03 18:05:11.534195 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:11.534159    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/806db06a-f8d1-43cd-9a9a-452dbdc0c949-kube-api-access-jj74h\" (UniqueName: \"kubernetes.io/projected/806db06a-f8d1-43cd-9a9a-452dbdc0c949-kube-api-access-jj74h\") pod \"806db06a-f8d1-43cd-9a9a-452dbdc0c949\" (UID: \"806db06a-f8d1-43cd-9a9a-452dbdc0c949\") "
Nov 03 18:05:11.534507 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:11.534487    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/806db06a-f8d1-43cd-9a9a-452dbdc0c949-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/806db06a-f8d1-43cd-9a9a-452dbdc0c949-maas-api-openshift-ca\") pod \"806db06a-f8d1-43cd-9a9a-452dbdc0c949\" (UID: \"806db06a-f8d1-43cd-9a9a-452dbdc0c949\") "
Nov 03 18:05:11.534738 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:11.534719    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/806db06a-f8d1-43cd-9a9a-452dbdc0c949-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/806db06a-f8d1-43cd-9a9a-452dbdc0c949-maas-api-xks-ca\") pod \"806db06a-f8d1-43cd-9a9a-452dbdc0c949\" (UID: \"806db06a-f8d1-43cd-9a9a-452dbdc0c949\") "
Nov 03 18:05:11.536333 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:11.536308    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/806db06a-f8d1-43cd-9a9a-452dbdc0c949-maas-api-openshift-ca" pod "806db06a-f8d1-43cd-9a9a-452dbdc0c949" (UID: "806db06a-f8d1-43cd-9a9a-452dbdc0c949"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:05:11.540991 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:11.540964    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/806db06a-f8d1-43cd-9a9a-452dbdc0c949-kube-api-access-jj74h" pod "806db06a-f8d1-43cd-9a9a-452dbdc0c949" (UID: "806db06a-f8d1-43cd-9a9a-452dbdc0c949"). InnerVolumeSpecName "kube-api-access-jj74h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:05:11.612955 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:11.600962    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/806db06a-f8d1-43cd-9a9a-452dbdc0c949-maas-api-xks-ca" pod "806db06a-f8d1-43cd-9a9a-452dbdc0c949" (UID: "806db06a-f8d1-43cd-9a9a-452dbdc0c949"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:05:11.643628 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:11.643360    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj74h\" (UniqueName: \"kubernetes.io/projected/806db06a-f8d1-43cd-9a9a-452dbdc0c949-kube-api-access-jj74h\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:05:11.643628 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:11.643397    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/806db06a-f8d1-43cd-9a9a-452dbdc0c949-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:05:11.643628 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:11.643415    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/806db06a-f8d1-43cd-9a9a-452dbdc0c949-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:05:12.083663 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.081804    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479"]
Nov 03 18:05:12.083663 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.082634    5315 cpu_manager.go:414] "containerMap: removing container" podUID="806db06a-f8d1-43cd-9a9a-452dbdc0c949" containerName="revoke-keys"
Nov 03 18:05:12.083663 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.082654    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="806db06a-f8d1-43cd-9a9a-452dbdc0c949" containerName="revoke-keys"
Nov 03 18:05:12.083663 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.082678    5315 cpu_manager.go:414] "containerMap: removing container" podUID="23d87d88-8b0e-4e14-9fa0-cc704c31e415" containerName="revoke-keys"
Nov 03 18:05:12.083663 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.082706    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23d87d88-8b0e-4e14-9fa0-cc704c31e415" containerName="revoke-keys"
Nov 03 18:05:12.083663 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.082862    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="23d87d88-8b0e-4e14-9fa0-cc704c31e415" containerName="revoke-keys"
Nov 03 18:05:12.083663 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.082885    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="806db06a-f8d1-43cd-9a9a-452dbdc0c949" containerName="revoke-keys"
Nov 03 18:05:12.096633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.092431    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479"
Nov 03 18:05:12.186635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.186525    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479"]
Nov 03 18:05:12.423708 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.332410    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-credential-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479\" (UID: \"a23b3a10-d0ee-4eee-802e-97040ba5bd9d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479"
Nov 03 18:05:12.423708 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.332502    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-workload-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479\" (UID: \"a23b3a10-d0ee-4eee-802e-97040ba5bd9d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479"
Nov 03 18:05:12.423708 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.332556    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-istio-data\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479\" (UID: \"a23b3a10-d0ee-4eee-802e-97040ba5bd9d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479"
Nov 03 18:05:12.423708 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.332634    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-istio-envoy\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479\" (UID: \"a23b3a10-d0ee-4eee-802e-97040ba5bd9d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479"
Nov 03 18:05:12.423708 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.332685    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-istio-token\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479\" (UID: \"a23b3a10-d0ee-4eee-802e-97040ba5bd9d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479"
Nov 03 18:05:12.423708 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.332740    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-workload-certs\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479\" (UID: \"a23b3a10-d0ee-4eee-802e-97040ba5bd9d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479"
Nov 03 18:05:12.423708 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.332865    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-istiod-ca-cert\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479\" (UID: \"a23b3a10-d0ee-4eee-802e-97040ba5bd9d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479"
Nov 03 18:05:12.423708 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.332906    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jncnn\" (UniqueName: \"kubernetes.io/projected/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-kube-api-access-jncnn\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479\" (UID: \"a23b3a10-d0ee-4eee-802e-97040ba5bd9d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479"
Nov 03 18:05:12.423708 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.332946    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-istio-podinfo\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479\" (UID: \"a23b3a10-d0ee-4eee-802e-97040ba5bd9d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479"
Nov 03 18:05:12.423708 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.355032    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e4da69-lqlb9" event={"ID":"806db06a-f8d1-43cd-9a9a-452dbdc0c949","Type":"ContainerDied","Data":"23382d57a1ab2f1a257b0b7755bfb65ead45038cabc567032f95ecb950b42850"}
Nov 03 18:05:12.423708 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.355091    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23382d57a1ab2f1a257b0b7755bfb65ead45038cabc567032f95ecb950b42850"
Nov 03 18:05:12.423708 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.355165    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e4da69-lqlb9"
Nov 03 18:05:12.449695 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.448530    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-istio-token\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479\" (UID: \"a23b3a10-d0ee-4eee-802e-97040ba5bd9d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479"
Nov 03 18:05:12.449695 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.448649    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-workload-certs\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479\" (UID: \"a23b3a10-d0ee-4eee-802e-97040ba5bd9d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479"
Nov 03 18:05:12.449695 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.448772    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-istiod-ca-cert\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479\" (UID: \"a23b3a10-d0ee-4eee-802e-97040ba5bd9d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479"
Nov 03 18:05:12.449695 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.448818    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jncnn\" (UniqueName: \"kubernetes.io/projected/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-kube-api-access-jncnn\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479\" (UID: \"a23b3a10-d0ee-4eee-802e-97040ba5bd9d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479"
Nov 03 18:05:12.449695 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.448868    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-istio-podinfo\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479\" (UID: \"a23b3a10-d0ee-4eee-802e-97040ba5bd9d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479"
Nov 03 18:05:12.449695 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.448912    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-credential-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479\" (UID: \"a23b3a10-d0ee-4eee-802e-97040ba5bd9d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479"
Nov 03 18:05:12.449695 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.448970    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-workload-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479\" (UID: \"a23b3a10-d0ee-4eee-802e-97040ba5bd9d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479"
Nov 03 18:05:12.449695 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.449037    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-istio-data\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479\" (UID: \"a23b3a10-d0ee-4eee-802e-97040ba5bd9d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479"
Nov 03 18:05:12.449695 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.449098    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-istio-envoy\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479\" (UID: \"a23b3a10-d0ee-4eee-802e-97040ba5bd9d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479"
Nov 03 18:05:12.459636 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.457852    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-workload-certs\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479\" (UID: \"a23b3a10-d0ee-4eee-802e-97040ba5bd9d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479"
Nov 03 18:05:12.459636 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.458498    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-istiod-ca-cert\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479\" (UID: \"a23b3a10-d0ee-4eee-802e-97040ba5bd9d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479"
Nov 03 18:05:12.486281 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.483230    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-workload-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479\" (UID: \"a23b3a10-d0ee-4eee-802e-97040ba5bd9d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479"
Nov 03 18:05:12.486281 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.483552    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-credential-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479\" (UID: \"a23b3a10-d0ee-4eee-802e-97040ba5bd9d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479"
Nov 03 18:05:12.486281 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.484773    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-istio-data\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479\" (UID: \"a23b3a10-d0ee-4eee-802e-97040ba5bd9d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479"
Nov 03 18:05:12.516469 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.516414    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-istio-envoy\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479\" (UID: \"a23b3a10-d0ee-4eee-802e-97040ba5bd9d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479"
Nov 03 18:05:12.557872 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.557835    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jncnn\" (UniqueName: \"kubernetes.io/projected/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-kube-api-access-jncnn\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479\" (UID: \"a23b3a10-d0ee-4eee-802e-97040ba5bd9d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479"
Nov 03 18:05:12.558690 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.558602    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-istio-token\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479\" (UID: \"a23b3a10-d0ee-4eee-802e-97040ba5bd9d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479"
Nov 03 18:05:12.606661 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.600723    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-istio-podinfo\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479\" (UID: \"a23b3a10-d0ee-4eee-802e-97040ba5bd9d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479"
Nov 03 18:05:12.609709 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.609673    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb"]
Nov 03 18:05:12.618368 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.618338    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb"]
Nov 03 18:05:12.618692 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.618663    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb"
Nov 03 18:05:12.692200 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.692087    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-workload-certs\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb\" (UID: \"858e4f16-d2d2-4ba6-9e75-b81fe8ad5674\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb"
Nov 03 18:05:12.692529 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.692501    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-istio-podinfo\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb\" (UID: \"858e4f16-d2d2-4ba6-9e75-b81fe8ad5674\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb"
Nov 03 18:05:12.692752 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.692733    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-istio-envoy\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb\" (UID: \"858e4f16-d2d2-4ba6-9e75-b81fe8ad5674\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb"
Nov 03 18:05:12.692935 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.692908    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-istio-data\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb\" (UID: \"858e4f16-d2d2-4ba6-9e75-b81fe8ad5674\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb"
Nov 03 18:05:12.693113 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.693093    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-credential-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb\" (UID: \"858e4f16-d2d2-4ba6-9e75-b81fe8ad5674\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb"
Nov 03 18:05:12.693389 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.693369    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-workload-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb\" (UID: \"858e4f16-d2d2-4ba6-9e75-b81fe8ad5674\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb"
Nov 03 18:05:12.693544 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.693525    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lskdx\" (UniqueName: \"kubernetes.io/projected/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-kube-api-access-lskdx\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb\" (UID: \"858e4f16-d2d2-4ba6-9e75-b81fe8ad5674\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb"
Nov 03 18:05:12.693789 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.693769    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-istiod-ca-cert\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb\" (UID: \"858e4f16-d2d2-4ba6-9e75-b81fe8ad5674\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb"
Nov 03 18:05:12.694013 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.693990    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-istio-token\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb\" (UID: \"858e4f16-d2d2-4ba6-9e75-b81fe8ad5674\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb"
Nov 03 18:05:12.723452 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.723048    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479"
Nov 03 18:05:12.805494 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.803455    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-istio-podinfo\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb\" (UID: \"858e4f16-d2d2-4ba6-9e75-b81fe8ad5674\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb"
Nov 03 18:05:12.805494 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.803527    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-istio-envoy\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb\" (UID: \"858e4f16-d2d2-4ba6-9e75-b81fe8ad5674\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb"
Nov 03 18:05:12.805494 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.803571    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-istio-data\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb\" (UID: \"858e4f16-d2d2-4ba6-9e75-b81fe8ad5674\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb"
Nov 03 18:05:12.805494 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.803602    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-credential-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb\" (UID: \"858e4f16-d2d2-4ba6-9e75-b81fe8ad5674\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb"
Nov 03 18:05:12.805494 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.803711    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-workload-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb\" (UID: \"858e4f16-d2d2-4ba6-9e75-b81fe8ad5674\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb"
Nov 03 18:05:12.805494 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.803739    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lskdx\" (UniqueName: \"kubernetes.io/projected/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-kube-api-access-lskdx\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb\" (UID: \"858e4f16-d2d2-4ba6-9e75-b81fe8ad5674\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb"
Nov 03 18:05:12.805494 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.803796    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-istiod-ca-cert\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb\" (UID: \"858e4f16-d2d2-4ba6-9e75-b81fe8ad5674\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb"
Nov 03 18:05:12.805494 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.803874    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-istio-token\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb\" (UID: \"858e4f16-d2d2-4ba6-9e75-b81fe8ad5674\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb"
Nov 03 18:05:12.805494 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.803927    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-workload-certs\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb\" (UID: \"858e4f16-d2d2-4ba6-9e75-b81fe8ad5674\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb"
Nov 03 18:05:12.805494 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.804440    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-workload-certs\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb\" (UID: \"858e4f16-d2d2-4ba6-9e75-b81fe8ad5674\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb"
Nov 03 18:05:12.805494 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.805122    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-workload-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb\" (UID: \"858e4f16-d2d2-4ba6-9e75-b81fe8ad5674\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb"
Nov 03 18:05:12.806268 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.805797    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-istio-data\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb\" (UID: \"858e4f16-d2d2-4ba6-9e75-b81fe8ad5674\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb"
Nov 03 18:05:12.806403 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.806370    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-credential-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb\" (UID: \"858e4f16-d2d2-4ba6-9e75-b81fe8ad5674\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb"
Nov 03 18:05:12.807365 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.807335    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-istiod-ca-cert\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb\" (UID: \"858e4f16-d2d2-4ba6-9e75-b81fe8ad5674\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb"
Nov 03 18:05:12.819537 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.818294    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-istio-podinfo\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb\" (UID: \"858e4f16-d2d2-4ba6-9e75-b81fe8ad5674\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb"
Nov 03 18:05:12.821580 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.821482    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-istio-envoy\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb\" (UID: \"858e4f16-d2d2-4ba6-9e75-b81fe8ad5674\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb"
Nov 03 18:05:12.888633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.885414    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lskdx\" (UniqueName: \"kubernetes.io/projected/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-kube-api-access-lskdx\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb\" (UID: \"858e4f16-d2d2-4ba6-9e75-b81fe8ad5674\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb"
Nov 03 18:05:12.906632 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.903302    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-istio-token\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb\" (UID: \"858e4f16-d2d2-4ba6-9e75-b81fe8ad5674\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb"
Nov 03 18:05:12.941303 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:12.940467    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb"
Nov 03 18:05:13.683653 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:13.681850    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479"]
Nov 03 18:05:13.701670 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:13.701626    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:05:13.701961 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:13.701899    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:05:13.702152 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:13.702131    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:05:14.053727 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:14.053678    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb"]
Nov 03 18:05:14.058656 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:05:14.058626    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod858e4f16_d2d2_4ba6_9e75_b81fe8ad5674.slice/crio-1f87bfb2af9e537420c9baf9ed14bf092911aa1cd3b30ab69b32c6300a425a2b WatchSource:0}: Error finding container 1f87bfb2af9e537420c9baf9ed14bf092911aa1cd3b30ab69b32c6300a425a2b: Status 404 returned error can't find the container with id 1f87bfb2af9e537420c9baf9ed14bf092911aa1cd3b30ab69b32c6300a425a2b
Nov 03 18:05:14.061453 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:14.061406    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:05:14.061573 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:14.061481    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:05:14.061573 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:14.061513    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:05:14.604909 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:14.604863    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb" event={"ID":"858e4f16-d2d2-4ba6-9e75-b81fe8ad5674","Type":"ContainerStarted","Data":"b9d849e23423046d448c22db5a82cbfa8eda6c1ae42c10d53a41141d36404331"}
Nov 03 18:05:14.605118 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:14.604938    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb" event={"ID":"858e4f16-d2d2-4ba6-9e75-b81fe8ad5674","Type":"ContainerStarted","Data":"1f87bfb2af9e537420c9baf9ed14bf092911aa1cd3b30ab69b32c6300a425a2b"}
Nov 03 18:05:14.606940 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:14.606914    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479" event={"ID":"a23b3a10-d0ee-4eee-802e-97040ba5bd9d","Type":"ContainerStarted","Data":"75bf9b7fbbdfe8b707815163673abbbac96dacde2fdec6cbb3e8ad4c227fac5c"}
Nov 03 18:05:14.607084 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:14.606954    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479" event={"ID":"a23b3a10-d0ee-4eee-802e-97040ba5bd9d","Type":"ContainerStarted","Data":"a252e91cc245d28f00a21fd7d6dce60a4e15c505944c88ab2f1b9c93c151f918"}
Nov 03 18:05:14.723555 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:14.723518    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479"
Nov 03 18:05:14.739657 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:14.727775    5315 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 03 18:05:14.739657 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:14.727858    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479" podUID="a23b3a10-d0ee-4eee-802e-97040ba5bd9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 03 18:05:14.739657 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:14.736819    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb" podStartSLOduration=2.736797437 podStartE2EDuration="2.736797437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:05:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:05:14.723222904 +0000 UTC m=+1153.739420998" watchObservedRunningTime="2026-11-03 18:05:14.736797437 +0000 UTC m=+1153.752995506"
Nov 03 18:05:14.847632 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:14.844509    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479" podStartSLOduration=2.8444870719999997 podStartE2EDuration="2.844487072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:05:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:05:14.84075487 +0000 UTC m=+1153.856952953" watchObservedRunningTime="2026-11-03 18:05:14.844487072 +0000 UTC m=+1153.860685142"
Nov 03 18:05:14.885836 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:14.885743    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75071396-882t2"]
Nov 03 18:05:14.891846 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:14.891819    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75071396-882t2"
Nov 03 18:05:14.950709 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:14.950671    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:05:14.951029 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:14.951010    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:05:14.951191 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:14.951178    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:05:14.951386 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:14.951372    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:05:14.970632 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:14.970570    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb"
Nov 03 18:05:14.973357 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:14.973327    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75071396-882t2"]
Nov 03 18:05:14.973705 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:14.973573    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d82ee37-166a-4433-9733-1c9f55d6d534-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75071396-882t2\" (UID: \"6d82ee37-166a-4433-9733-1c9f55d6d534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75071396-882t2"
Nov 03 18:05:14.973970 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:14.973951    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghdf4\" (UniqueName: \"kubernetes.io/projected/6d82ee37-166a-4433-9733-1c9f55d6d534-kube-api-access-ghdf4\") pod \"maas-api-revoke-sub-75071396-882t2\" (UID: \"6d82ee37-166a-4433-9733-1c9f55d6d534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75071396-882t2"
Nov 03 18:05:14.974127 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:14.974107    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d82ee37-166a-4433-9733-1c9f55d6d534-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75071396-882t2\" (UID: \"6d82ee37-166a-4433-9733-1c9f55d6d534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75071396-882t2"
Nov 03 18:05:15.087507 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:15.087454    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d82ee37-166a-4433-9733-1c9f55d6d534-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75071396-882t2\" (UID: \"6d82ee37-166a-4433-9733-1c9f55d6d534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75071396-882t2"
Nov 03 18:05:15.087737 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:15.087563    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghdf4\" (UniqueName: \"kubernetes.io/projected/6d82ee37-166a-4433-9733-1c9f55d6d534-kube-api-access-ghdf4\") pod \"maas-api-revoke-sub-75071396-882t2\" (UID: \"6d82ee37-166a-4433-9733-1c9f55d6d534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75071396-882t2"
Nov 03 18:05:15.087737 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:15.087640    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d82ee37-166a-4433-9733-1c9f55d6d534-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75071396-882t2\" (UID: \"6d82ee37-166a-4433-9733-1c9f55d6d534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75071396-882t2"
Nov 03 18:05:15.088497 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:15.088474    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d82ee37-166a-4433-9733-1c9f55d6d534-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75071396-882t2\" (UID: \"6d82ee37-166a-4433-9733-1c9f55d6d534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75071396-882t2"
Nov 03 18:05:15.090747 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:15.090719    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d82ee37-166a-4433-9733-1c9f55d6d534-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75071396-882t2\" (UID: \"6d82ee37-166a-4433-9733-1c9f55d6d534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75071396-882t2"
Nov 03 18:05:15.179748 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:15.168958    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghdf4\" (UniqueName: \"kubernetes.io/projected/6d82ee37-166a-4433-9733-1c9f55d6d534-kube-api-access-ghdf4\") pod \"maas-api-revoke-sub-75071396-882t2\" (UID: \"6d82ee37-166a-4433-9733-1c9f55d6d534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75071396-882t2"
Nov 03 18:05:15.214447 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:15.214207    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75071396-882t2"
Nov 03 18:05:15.694586 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:15.694552    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75071396-882t2"]
Nov 03 18:05:15.699508 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:05:15.699478    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d82ee37_166a_4433_9733_1c9f55d6d534.slice/crio-f0a53c3ec00f87de87edffc681aa63052c6c3231b49873c4f50bb51f0d2451fd WatchSource:0}: Error finding container f0a53c3ec00f87de87edffc681aa63052c6c3231b49873c4f50bb51f0d2451fd: Status 404 returned error can't find the container with id f0a53c3ec00f87de87edffc681aa63052c6c3231b49873c4f50bb51f0d2451fd
Nov 03 18:05:15.766636 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:15.757756    5315 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 03 18:05:15.766636 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:15.757813    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479" podUID="a23b3a10-d0ee-4eee-802e-97040ba5bd9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 03 18:05:15.966694 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:15.952717    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 03 18:05:15.966694 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:15.952797    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb" podUID="858e4f16-d2d2-4ba6-9e75-b81fe8ad5674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 03 18:05:16.622814 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:16.622525    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75071396-882t2" event={"ID":"6d82ee37-166a-4433-9733-1c9f55d6d534","Type":"ContainerStarted","Data":"c5aca8edb998fc3a0c1a965b45ceb7cd9b8c0dee998fece6dff7b8cdd3a7e80f"}
Nov 03 18:05:16.622814 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:16.622586    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75071396-882t2" event={"ID":"6d82ee37-166a-4433-9733-1c9f55d6d534","Type":"ContainerStarted","Data":"f0a53c3ec00f87de87edffc681aa63052c6c3231b49873c4f50bb51f0d2451fd"}
Nov 03 18:05:16.751637 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:16.751346    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75071396-882t2" podStartSLOduration=2.751326546 podStartE2EDuration="2.751326546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:05:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:05:16.748032834 +0000 UTC m=+1155.764230904" watchObservedRunningTime="2026-11-03 18:05:16.751326546 +0000 UTC m=+1155.767524616"
Nov 03 18:05:16.774636 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:16.769742    5315 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 03 18:05:16.774636 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:16.769811    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479" podUID="a23b3a10-d0ee-4eee-802e-97040ba5bd9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 03 18:05:16.947310 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:16.944058    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 03 18:05:16.947310 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:16.944126    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb" podUID="858e4f16-d2d2-4ba6-9e75-b81fe8ad5674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 03 18:05:17.677254 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:17.677215    5315 generic.go:356] "Generic (PLEG): container finished" podID="6d82ee37-166a-4433-9733-1c9f55d6d534" containerID="c5aca8edb998fc3a0c1a965b45ceb7cd9b8c0dee998fece6dff7b8cdd3a7e80f" exitCode=0
Nov 03 18:05:17.677473 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:17.677383    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75071396-882t2" event={"ID":"6d82ee37-166a-4433-9733-1c9f55d6d534","Type":"ContainerDied","Data":"c5aca8edb998fc3a0c1a965b45ceb7cd9b8c0dee998fece6dff7b8cdd3a7e80f"}
Nov 03 18:05:17.745636 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:17.744800    5315 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 03 18:05:17.745636 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:17.744864    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479" podUID="a23b3a10-d0ee-4eee-802e-97040ba5bd9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 03 18:05:17.947490 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:17.947392    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 03 18:05:17.948060 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:17.948028    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb" podUID="858e4f16-d2d2-4ba6-9e75-b81fe8ad5674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 03 18:05:18.745116 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:18.744219    5315 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 03 18:05:18.745116 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:18.744286    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479" podUID="a23b3a10-d0ee-4eee-802e-97040ba5bd9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 03 18:05:18.919075 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:18.919047    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75071396-882t2"
Nov 03 18:05:18.977300 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:18.977191    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 03 18:05:18.977300 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:18.977253    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb" podUID="858e4f16-d2d2-4ba6-9e75-b81fe8ad5674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 03 18:05:19.094427 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:19.093929    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d82ee37-166a-4433-9733-1c9f55d6d534-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d82ee37-166a-4433-9733-1c9f55d6d534-maas-api-openshift-ca\") pod \"6d82ee37-166a-4433-9733-1c9f55d6d534\" (UID: \"6d82ee37-166a-4433-9733-1c9f55d6d534\") "
Nov 03 18:05:19.094427 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:19.094044    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d82ee37-166a-4433-9733-1c9f55d6d534-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d82ee37-166a-4433-9733-1c9f55d6d534-maas-api-xks-ca\") pod \"6d82ee37-166a-4433-9733-1c9f55d6d534\" (UID: \"6d82ee37-166a-4433-9733-1c9f55d6d534\") "
Nov 03 18:05:19.094427 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:19.094143    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d82ee37-166a-4433-9733-1c9f55d6d534-kube-api-access-ghdf4\" (UniqueName: \"kubernetes.io/projected/6d82ee37-166a-4433-9733-1c9f55d6d534-kube-api-access-ghdf4\") pod \"6d82ee37-166a-4433-9733-1c9f55d6d534\" (UID: \"6d82ee37-166a-4433-9733-1c9f55d6d534\") "
Nov 03 18:05:19.094800 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:19.094761    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d82ee37-166a-4433-9733-1c9f55d6d534-maas-api-openshift-ca" pod "6d82ee37-166a-4433-9733-1c9f55d6d534" (UID: "6d82ee37-166a-4433-9733-1c9f55d6d534"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:05:19.099581 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:19.099118    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d82ee37-166a-4433-9733-1c9f55d6d534-kube-api-access-ghdf4" pod "6d82ee37-166a-4433-9733-1c9f55d6d534" (UID: "6d82ee37-166a-4433-9733-1c9f55d6d534"). InnerVolumeSpecName "kube-api-access-ghdf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:05:19.099739 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:19.099718    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d82ee37-166a-4433-9733-1c9f55d6d534-maas-api-xks-ca" pod "6d82ee37-166a-4433-9733-1c9f55d6d534" (UID: "6d82ee37-166a-4433-9733-1c9f55d6d534"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:05:19.195692 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:19.195661    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d82ee37-166a-4433-9733-1c9f55d6d534-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:05:19.195692 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:19.195690    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d82ee37-166a-4433-9733-1c9f55d6d534-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:05:19.195692 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:19.195699    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghdf4\" (UniqueName: \"kubernetes.io/projected/6d82ee37-166a-4433-9733-1c9f55d6d534-kube-api-access-ghdf4\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:05:19.692338 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:19.692302    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75071396-882t2"
Nov 03 18:05:19.692511 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:19.692312    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75071396-882t2" event={"ID":"6d82ee37-166a-4433-9733-1c9f55d6d534","Type":"ContainerDied","Data":"f0a53c3ec00f87de87edffc681aa63052c6c3231b49873c4f50bb51f0d2451fd"}
Nov 03 18:05:19.692511 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:19.692436    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0a53c3ec00f87de87edffc681aa63052c6c3231b49873c4f50bb51f0d2451fd"
Nov 03 18:05:19.750634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:19.750263    5315 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 03 18:05:19.750634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:19.750329    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479" podUID="a23b3a10-d0ee-4eee-802e-97040ba5bd9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 03 18:05:19.945693 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:19.944695    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 03 18:05:19.945693 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:19.944756    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb" podUID="858e4f16-d2d2-4ba6-9e75-b81fe8ad5674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 03 18:05:20.330029 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:20.329987    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ff86cc4-mgz25"]
Nov 03 18:05:20.330961 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:20.330932    5315 cpu_manager.go:414] "containerMap: removing container" podUID="6d82ee37-166a-4433-9733-1c9f55d6d534" containerName="revoke-keys"
Nov 03 18:05:20.330961 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:20.330959    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d82ee37-166a-4433-9733-1c9f55d6d534" containerName="revoke-keys"
Nov 03 18:05:20.331170 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:20.331127    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d82ee37-166a-4433-9733-1c9f55d6d534" containerName="revoke-keys"
Nov 03 18:05:20.335323 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:20.335289    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff86cc4-mgz25"
Nov 03 18:05:20.384040 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:20.384009    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:05:20.384240 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:20.384213    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:05:20.384240 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:20.384058    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:05:20.384436 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:20.384361    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:05:20.392310 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:20.392275    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ff86cc4-mgz25"]
Nov 03 18:05:20.415406 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:20.415364    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ff86cc4-mgz25\" (UID: \"c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff86cc4-mgz25"
Nov 03 18:05:20.415406 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:20.415414    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4hz7\" (UniqueName: \"kubernetes.io/projected/c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e-kube-api-access-x4hz7\") pod \"maas-api-revoke-sub-5ff86cc4-mgz25\" (UID: \"c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff86cc4-mgz25"
Nov 03 18:05:20.415685 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:20.415627    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ff86cc4-mgz25\" (UID: \"c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff86cc4-mgz25"
Nov 03 18:05:20.517465 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:20.517421    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ff86cc4-mgz25\" (UID: \"c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff86cc4-mgz25"
Nov 03 18:05:20.517685 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:20.517471    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4hz7\" (UniqueName: \"kubernetes.io/projected/c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e-kube-api-access-x4hz7\") pod \"maas-api-revoke-sub-5ff86cc4-mgz25\" (UID: \"c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff86cc4-mgz25"
Nov 03 18:05:20.517685 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:20.517636    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ff86cc4-mgz25\" (UID: \"c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff86cc4-mgz25"
Nov 03 18:05:20.518285 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:20.518261    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ff86cc4-mgz25\" (UID: \"c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff86cc4-mgz25"
Nov 03 18:05:20.520211 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:20.520180    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ff86cc4-mgz25\" (UID: \"c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff86cc4-mgz25"
Nov 03 18:05:20.575233 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:20.575198    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4hz7\" (UniqueName: \"kubernetes.io/projected/c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e-kube-api-access-x4hz7\") pod \"maas-api-revoke-sub-5ff86cc4-mgz25\" (UID: \"c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff86cc4-mgz25"
Nov 03 18:05:20.653564 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:20.653475    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff86cc4-mgz25"
Nov 03 18:05:20.724646 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:20.724141    5315 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 03 18:05:20.724646 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:20.724214    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479" podUID="a23b3a10-d0ee-4eee-802e-97040ba5bd9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 03 18:05:20.941356 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:20.941260    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 03 18:05:20.941356 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:20.941322    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb" podUID="858e4f16-d2d2-4ba6-9e75-b81fe8ad5674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 03 18:05:21.064589 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:21.064537    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ff86cc4-mgz25"]
Nov 03 18:05:21.070377 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:05:21.070317    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc23e6e4d_f0f9_4fc9_8d09_d2e9a9a5503e.slice/crio-b828164e8168e96409ce48f83be893077a84d376da0895b4aacf57390248aebf WatchSource:0}: Error finding container b828164e8168e96409ce48f83be893077a84d376da0895b4aacf57390248aebf: Status 404 returned error can't find the container with id b828164e8168e96409ce48f83be893077a84d376da0895b4aacf57390248aebf
Nov 03 18:05:21.754275 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:21.750978    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479"
Nov 03 18:05:21.754275 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:21.752173    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479"
Nov 03 18:05:21.754275 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:21.753797    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479"
Nov 03 18:05:21.759642 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:21.757908    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff86cc4-mgz25" event={"ID":"c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e","Type":"ContainerStarted","Data":"e9094333e388d4000ebec138eb6815cc2c64f4b06b382afd9596eb9a2da39d79"}
Nov 03 18:05:21.759642 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:21.757965    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff86cc4-mgz25" event={"ID":"c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e","Type":"ContainerStarted","Data":"b828164e8168e96409ce48f83be893077a84d376da0895b4aacf57390248aebf"}
Nov 03 18:05:21.949633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:21.947915    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb"
Nov 03 18:05:21.949633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:21.949162    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb"
Nov 03 18:05:21.956283 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:21.956256    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb"
Nov 03 18:05:21.984688 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:21.984590    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff86cc4-mgz25" podStartSLOduration=1.984572586 podStartE2EDuration="1.984572586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:05:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:05:21.984164482 +0000 UTC m=+1161.000362551" watchObservedRunningTime="2026-11-03 18:05:21.984572586 +0000 UTC m=+1161.000770656"
Nov 03 18:05:22.063720 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:22.063657    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50b3a56f-zf47h"]
Nov 03 18:05:22.069546 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:22.069506    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50b3a56f-zf47h"]
Nov 03 18:05:22.775574 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:22.775540    5315 generic.go:356] "Generic (PLEG): container finished" podID="c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e" containerID="e9094333e388d4000ebec138eb6815cc2c64f4b06b382afd9596eb9a2da39d79" exitCode=0
Nov 03 18:05:22.776073 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:22.775653    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff86cc4-mgz25" event={"ID":"c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e","Type":"ContainerDied","Data":"e9094333e388d4000ebec138eb6815cc2c64f4b06b382afd9596eb9a2da39d79"}
Nov 03 18:05:23.108945 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:23.108903    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c493b7e-6578-4e59-bcd5-46e7e477a1f0" path="/var/lib/kubelet/pods/4c493b7e-6578-4e59-bcd5-46e7e477a1f0/volumes"
Nov 03 18:05:23.923075 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:23.923048    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff86cc4-mgz25"
Nov 03 18:05:24.017018 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:24.016978    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e-maas-api-xks-ca\") pod \"c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e\" (UID: \"c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e\") "
Nov 03 18:05:24.017185 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:24.017030    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e-maas-api-openshift-ca\") pod \"c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e\" (UID: \"c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e\") "
Nov 03 18:05:24.017185 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:24.017057    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e-kube-api-access-x4hz7\" (UniqueName: \"kubernetes.io/projected/c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e-kube-api-access-x4hz7\") pod \"c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e\" (UID: \"c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e\") "
Nov 03 18:05:24.017432 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:24.017402    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e-maas-api-openshift-ca" pod "c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e" (UID: "c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:05:24.017576 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:24.017555    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:05:24.019135 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:24.019112    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e-maas-api-xks-ca" pod "c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e" (UID: "c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:05:24.019208 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:24.019177    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e-kube-api-access-x4hz7" pod "c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e" (UID: "c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e"). InnerVolumeSpecName "kube-api-access-x4hz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:05:24.118718 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:24.118637    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:05:24.118718 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:24.118667    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4hz7\" (UniqueName: \"kubernetes.io/projected/c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e-kube-api-access-x4hz7\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:05:24.577998 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:05:24.577960    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:05:24.790919 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:24.790889    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff86cc4-mgz25"
Nov 03 18:05:24.791096 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:24.790933    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff86cc4-mgz25" event={"ID":"c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e","Type":"ContainerDied","Data":"b828164e8168e96409ce48f83be893077a84d376da0895b4aacf57390248aebf"}
Nov 03 18:05:24.791096 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:24.790990    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b828164e8168e96409ce48f83be893077a84d376da0895b4aacf57390248aebf"
Nov 03 18:05:26.522664 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:26.522598    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn"]
Nov 03 18:05:26.523545 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:26.523521    5315 cpu_manager.go:414] "containerMap: removing container" podUID="c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e" containerName="revoke-keys"
Nov 03 18:05:26.523545 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:26.523545    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e" containerName="revoke-keys"
Nov 03 18:05:26.523756 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:26.523729    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e" containerName="revoke-keys"
Nov 03 18:05:26.527524 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:26.527503    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn"
Nov 03 18:05:26.591381 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:26.591320    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn"]
Nov 03 18:05:26.648256 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:26.648214    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-credential-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn\" (UID: \"ab9a094b-19e7-4bf3-a4f6-63b953d47af2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn"
Nov 03 18:05:26.648541 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:26.648513    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-istio-token\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn\" (UID: \"ab9a094b-19e7-4bf3-a4f6-63b953d47af2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn"
Nov 03 18:05:26.648703 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:26.648573    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-istiod-ca-cert\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn\" (UID: \"ab9a094b-19e7-4bf3-a4f6-63b953d47af2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn"
Nov 03 18:05:26.648703 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:26.648636    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-istio-envoy\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn\" (UID: \"ab9a094b-19e7-4bf3-a4f6-63b953d47af2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn"
Nov 03 18:05:26.648703 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:26.648666    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-workload-certs\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn\" (UID: \"ab9a094b-19e7-4bf3-a4f6-63b953d47af2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn"
Nov 03 18:05:26.648935 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:26.648739    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-istio-podinfo\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn\" (UID: \"ab9a094b-19e7-4bf3-a4f6-63b953d47af2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn"
Nov 03 18:05:26.648935 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:26.648811    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-workload-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn\" (UID: \"ab9a094b-19e7-4bf3-a4f6-63b953d47af2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn"
Nov 03 18:05:26.648935 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:26.648845    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kglx\" (UniqueName: \"kubernetes.io/projected/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-kube-api-access-4kglx\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn\" (UID: \"ab9a094b-19e7-4bf3-a4f6-63b953d47af2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn"
Nov 03 18:05:26.648935 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:26.648892    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-istio-data\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn\" (UID: \"ab9a094b-19e7-4bf3-a4f6-63b953d47af2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn"
Nov 03 18:05:26.750141 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:26.750099    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-istio-podinfo\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn\" (UID: \"ab9a094b-19e7-4bf3-a4f6-63b953d47af2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn"
Nov 03 18:05:26.750346 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:26.750181    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-workload-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn\" (UID: \"ab9a094b-19e7-4bf3-a4f6-63b953d47af2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn"
Nov 03 18:05:26.750346 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:26.750214    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kglx\" (UniqueName: \"kubernetes.io/projected/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-kube-api-access-4kglx\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn\" (UID: \"ab9a094b-19e7-4bf3-a4f6-63b953d47af2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn"
Nov 03 18:05:26.750346 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:26.750270    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-istio-data\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn\" (UID: \"ab9a094b-19e7-4bf3-a4f6-63b953d47af2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn"
Nov 03 18:05:26.750346 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:26.750309    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-credential-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn\" (UID: \"ab9a094b-19e7-4bf3-a4f6-63b953d47af2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn"
Nov 03 18:05:26.750559 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:26.750399    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-istio-token\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn\" (UID: \"ab9a094b-19e7-4bf3-a4f6-63b953d47af2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn"
Nov 03 18:05:26.750559 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:26.750446    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-istiod-ca-cert\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn\" (UID: \"ab9a094b-19e7-4bf3-a4f6-63b953d47af2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn"
Nov 03 18:05:26.750559 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:26.750497    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-istio-envoy\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn\" (UID: \"ab9a094b-19e7-4bf3-a4f6-63b953d47af2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn"
Nov 03 18:05:26.750559 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:26.750528    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-workload-certs\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn\" (UID: \"ab9a094b-19e7-4bf3-a4f6-63b953d47af2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn"
Nov 03 18:05:26.750851 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:26.750641    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-workload-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn\" (UID: \"ab9a094b-19e7-4bf3-a4f6-63b953d47af2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn"
Nov 03 18:05:26.750851 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:26.750724    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-credential-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn\" (UID: \"ab9a094b-19e7-4bf3-a4f6-63b953d47af2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn"
Nov 03 18:05:26.750960 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:26.750883    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-istio-data\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn\" (UID: \"ab9a094b-19e7-4bf3-a4f6-63b953d47af2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn"
Nov 03 18:05:26.751012 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:26.750975    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-workload-certs\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn\" (UID: \"ab9a094b-19e7-4bf3-a4f6-63b953d47af2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn"
Nov 03 18:05:26.751301 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:26.751273    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-istiod-ca-cert\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn\" (UID: \"ab9a094b-19e7-4bf3-a4f6-63b953d47af2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn"
Nov 03 18:05:26.752683 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:26.752661    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-istio-envoy\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn\" (UID: \"ab9a094b-19e7-4bf3-a4f6-63b953d47af2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn"
Nov 03 18:05:26.752796 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:26.752777    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-istio-podinfo\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn\" (UID: \"ab9a094b-19e7-4bf3-a4f6-63b953d47af2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn"
Nov 03 18:05:26.807877 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:26.807842    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-istio-token\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn\" (UID: \"ab9a094b-19e7-4bf3-a4f6-63b953d47af2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn"
Nov 03 18:05:26.808060 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:26.807908    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kglx\" (UniqueName: \"kubernetes.io/projected/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-kube-api-access-4kglx\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn\" (UID: \"ab9a094b-19e7-4bf3-a4f6-63b953d47af2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn"
Nov 03 18:05:26.841266 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:26.841232    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn"
Nov 03 18:05:26.851912 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:26.851879    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp"]
Nov 03 18:05:26.856253 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:26.856228    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp"
Nov 03 18:05:26.926693 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:26.923546    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp"]
Nov 03 18:05:26.953770 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:26.953737    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-workload-certs\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp\" (UID: \"3cf05bbd-6605-4cd2-aa94-85cbb46223aa\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp"
Nov 03 18:05:26.953921 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:26.953806    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-istiod-ca-cert\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp\" (UID: \"3cf05bbd-6605-4cd2-aa94-85cbb46223aa\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp"
Nov 03 18:05:26.953960 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:26.953929    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-istio-data\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp\" (UID: \"3cf05bbd-6605-4cd2-aa94-85cbb46223aa\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp"
Nov 03 18:05:26.954045 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:26.953961    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-workload-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp\" (UID: \"3cf05bbd-6605-4cd2-aa94-85cbb46223aa\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp"
Nov 03 18:05:26.954045 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:26.953984    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlh4w\" (UniqueName: \"kubernetes.io/projected/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-kube-api-access-jlh4w\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp\" (UID: \"3cf05bbd-6605-4cd2-aa94-85cbb46223aa\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp"
Nov 03 18:05:26.954125 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:26.954061    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-istio-token\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp\" (UID: \"3cf05bbd-6605-4cd2-aa94-85cbb46223aa\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp"
Nov 03 18:05:26.954158 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:26.954143    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-istio-envoy\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp\" (UID: \"3cf05bbd-6605-4cd2-aa94-85cbb46223aa\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp"
Nov 03 18:05:26.954241 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:26.954213    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-credential-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp\" (UID: \"3cf05bbd-6605-4cd2-aa94-85cbb46223aa\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp"
Nov 03 18:05:26.954379 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:26.954358    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-istio-podinfo\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp\" (UID: \"3cf05bbd-6605-4cd2-aa94-85cbb46223aa\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp"
Nov 03 18:05:27.055565 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:27.055533    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-istio-data\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp\" (UID: \"3cf05bbd-6605-4cd2-aa94-85cbb46223aa\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp"
Nov 03 18:05:27.055565 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:27.055575    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-workload-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp\" (UID: \"3cf05bbd-6605-4cd2-aa94-85cbb46223aa\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp"
Nov 03 18:05:27.055807 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:27.055712    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlh4w\" (UniqueName: \"kubernetes.io/projected/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-kube-api-access-jlh4w\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp\" (UID: \"3cf05bbd-6605-4cd2-aa94-85cbb46223aa\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp"
Nov 03 18:05:27.055807 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:27.055780    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-istio-token\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp\" (UID: \"3cf05bbd-6605-4cd2-aa94-85cbb46223aa\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp"
Nov 03 18:05:27.055977 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:27.055814    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-istio-envoy\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp\" (UID: \"3cf05bbd-6605-4cd2-aa94-85cbb46223aa\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp"
Nov 03 18:05:27.055977 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:27.055866    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-credential-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp\" (UID: \"3cf05bbd-6605-4cd2-aa94-85cbb46223aa\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp"
Nov 03 18:05:27.055977 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:27.055951    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-workload-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp\" (UID: \"3cf05bbd-6605-4cd2-aa94-85cbb46223aa\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp"
Nov 03 18:05:27.056270 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:27.055984    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-istio-data\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp\" (UID: \"3cf05bbd-6605-4cd2-aa94-85cbb46223aa\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp"
Nov 03 18:05:27.056270 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:27.055993    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-istio-podinfo\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp\" (UID: \"3cf05bbd-6605-4cd2-aa94-85cbb46223aa\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp"
Nov 03 18:05:27.056270 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:27.056061    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-workload-certs\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp\" (UID: \"3cf05bbd-6605-4cd2-aa94-85cbb46223aa\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp"
Nov 03 18:05:27.056270 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:27.056119    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-istiod-ca-cert\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp\" (UID: \"3cf05bbd-6605-4cd2-aa94-85cbb46223aa\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp"
Nov 03 18:05:27.056471 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:27.056323    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-credential-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp\" (UID: \"3cf05bbd-6605-4cd2-aa94-85cbb46223aa\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp"
Nov 03 18:05:27.056471 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:27.056452    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-workload-certs\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp\" (UID: \"3cf05bbd-6605-4cd2-aa94-85cbb46223aa\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp"
Nov 03 18:05:27.056956 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:27.056933    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-istiod-ca-cert\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp\" (UID: \"3cf05bbd-6605-4cd2-aa94-85cbb46223aa\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp"
Nov 03 18:05:27.058168 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:27.058118    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-istio-envoy\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp\" (UID: \"3cf05bbd-6605-4cd2-aa94-85cbb46223aa\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp"
Nov 03 18:05:27.058373 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:27.058357    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-istio-podinfo\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp\" (UID: \"3cf05bbd-6605-4cd2-aa94-85cbb46223aa\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp"
Nov 03 18:05:27.113362 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:27.113328    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-istio-token\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp\" (UID: \"3cf05bbd-6605-4cd2-aa94-85cbb46223aa\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp"
Nov 03 18:05:27.113495 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:27.113376    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlh4w\" (UniqueName: \"kubernetes.io/projected/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-kube-api-access-jlh4w\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp\" (UID: \"3cf05bbd-6605-4cd2-aa94-85cbb46223aa\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp"
Nov 03 18:05:27.180468 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:27.180432    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp"
Nov 03 18:05:27.280641 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:27.280586    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn"]
Nov 03 18:05:27.283589 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:27.283556    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:05:27.283706 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:27.283640    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:05:27.283706 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:27.283672    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:05:27.566579 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:27.566538    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp"]
Nov 03 18:05:27.571912 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:05:27.571849    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cf05bbd_6605_4cd2_aa94_85cbb46223aa.slice/crio-bfdea99b678db1b092e993d51d7b7f6da7baf8b67514794c92f5fa9189e5aa8f WatchSource:0}: Error finding container bfdea99b678db1b092e993d51d7b7f6da7baf8b67514794c92f5fa9189e5aa8f: Status 404 returned error can't find the container with id bfdea99b678db1b092e993d51d7b7f6da7baf8b67514794c92f5fa9189e5aa8f
Nov 03 18:05:27.574402 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:27.574362    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:05:27.574521 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:27.574433    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:05:27.574521 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:27.574463    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:05:27.813847 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:27.813800    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp" event={"ID":"3cf05bbd-6605-4cd2-aa94-85cbb46223aa","Type":"ContainerStarted","Data":"e6ac8fe445fba2f76fe442103c503bce99e901c53b27136bc55574863ff943d0"}
Nov 03 18:05:27.814016 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:27.813866    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp" event={"ID":"3cf05bbd-6605-4cd2-aa94-85cbb46223aa","Type":"ContainerStarted","Data":"bfdea99b678db1b092e993d51d7b7f6da7baf8b67514794c92f5fa9189e5aa8f"}
Nov 03 18:05:27.815590 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:27.815560    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn" event={"ID":"ab9a094b-19e7-4bf3-a4f6-63b953d47af2","Type":"ContainerStarted","Data":"6cfdec0dcdfd25265b87646dbbec47bf9ca3e9f2565de155f2e66c19c93ab698"}
Nov 03 18:05:27.815721 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:27.815628    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn" event={"ID":"ab9a094b-19e7-4bf3-a4f6-63b953d47af2","Type":"ContainerStarted","Data":"a4a24619592032e4702259ceee01c196cbe87123c2024ffc39d959d6b84bd23e"}
Nov 03 18:05:27.855861 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:27.855759    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn"
Nov 03 18:05:27.938958 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:27.925987    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp" podStartSLOduration=1.9259701219999998 podStartE2EDuration="1.925970122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:05:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:05:27.9254639 +0000 UTC m=+1166.941661971" watchObservedRunningTime="2026-11-03 18:05:27.925970122 +0000 UTC m=+1166.942168192"
Nov 03 18:05:28.182552 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:28.182451    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp"
Nov 03 18:05:28.187814 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:28.187776    5315 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 18:05:28.187912 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:28.187850    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp" podUID="3cf05bbd-6605-4cd2-aa94-85cbb46223aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 18:05:28.873843 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:28.873784    5315 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 03 18:05:28.874470 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:28.874431    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn" podUID="ab9a094b-19e7-4bf3-a4f6-63b953d47af2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 03 18:05:29.181637 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:29.181332    5315 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 18:05:29.181637 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:29.181389    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp" podUID="3cf05bbd-6605-4cd2-aa94-85cbb46223aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 18:05:29.571128 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:29.569349    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn" podStartSLOduration=3.569326206 podStartE2EDuration="3.569326206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:05:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:05:28.055060929 +0000 UTC m=+1167.071259005" watchObservedRunningTime="2026-11-03 18:05:29.569326206 +0000 UTC m=+1168.585524278"
Nov 03 18:05:29.577651 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:29.575431    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45755c76-x67jk"]
Nov 03 18:05:29.580117 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:29.580094    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45755c76-x67jk"
Nov 03 18:05:29.595177 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:29.595146    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pslvf\" (UniqueName: \"kubernetes.io/projected/2b23d8e6-0f73-4545-bbdf-e003b811018f-kube-api-access-pslvf\") pod \"maas-api-revoke-sub-45755c76-x67jk\" (UID: \"2b23d8e6-0f73-4545-bbdf-e003b811018f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45755c76-x67jk"
Nov 03 18:05:29.595313 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:29.595192    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b23d8e6-0f73-4545-bbdf-e003b811018f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45755c76-x67jk\" (UID: \"2b23d8e6-0f73-4545-bbdf-e003b811018f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45755c76-x67jk"
Nov 03 18:05:29.595397 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:29.595339    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b23d8e6-0f73-4545-bbdf-e003b811018f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45755c76-x67jk\" (UID: \"2b23d8e6-0f73-4545-bbdf-e003b811018f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45755c76-x67jk"
Nov 03 18:05:29.660631 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:29.659815    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45755c76-x67jk"]
Nov 03 18:05:29.660631 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:29.660147    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:05:29.660631 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:29.660286    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:05:29.660631 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:29.660381    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:05:29.683635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:29.661007    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:05:29.697658 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:29.697101    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b23d8e6-0f73-4545-bbdf-e003b811018f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45755c76-x67jk\" (UID: \"2b23d8e6-0f73-4545-bbdf-e003b811018f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45755c76-x67jk"
Nov 03 18:05:29.697658 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:29.697257    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pslvf\" (UniqueName: \"kubernetes.io/projected/2b23d8e6-0f73-4545-bbdf-e003b811018f-kube-api-access-pslvf\") pod \"maas-api-revoke-sub-45755c76-x67jk\" (UID: \"2b23d8e6-0f73-4545-bbdf-e003b811018f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45755c76-x67jk"
Nov 03 18:05:29.697658 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:29.697325    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b23d8e6-0f73-4545-bbdf-e003b811018f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45755c76-x67jk\" (UID: \"2b23d8e6-0f73-4545-bbdf-e003b811018f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45755c76-x67jk"
Nov 03 18:05:29.697951 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:29.697933    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b23d8e6-0f73-4545-bbdf-e003b811018f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45755c76-x67jk\" (UID: \"2b23d8e6-0f73-4545-bbdf-e003b811018f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45755c76-x67jk"
Nov 03 18:05:29.705303 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:29.705243    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b23d8e6-0f73-4545-bbdf-e003b811018f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45755c76-x67jk\" (UID: \"2b23d8e6-0f73-4545-bbdf-e003b811018f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45755c76-x67jk"
Nov 03 18:05:29.783856 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:29.783778    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pslvf\" (UniqueName: \"kubernetes.io/projected/2b23d8e6-0f73-4545-bbdf-e003b811018f-kube-api-access-pslvf\") pod \"maas-api-revoke-sub-45755c76-x67jk\" (UID: \"2b23d8e6-0f73-4545-bbdf-e003b811018f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45755c76-x67jk"
Nov 03 18:05:29.874694 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:29.874087    5315 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 03 18:05:29.874694 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:29.874162    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn" podUID="ab9a094b-19e7-4bf3-a4f6-63b953d47af2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 03 18:05:29.895912 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:29.895882    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45755c76-x67jk"
Nov 03 18:05:30.188669 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:30.184522    5315 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 18:05:30.188669 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:30.184591    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp" podUID="3cf05bbd-6605-4cd2-aa94-85cbb46223aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 18:05:30.357091 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:30.343659    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45755c76-x67jk"]
Nov 03 18:05:30.546485 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:30.546446    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a83934d6-nzswt"]
Nov 03 18:05:30.560148 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:30.560116    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83934d6-nzswt"
Nov 03 18:05:30.594363 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:30.594332    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a83934d6-nzswt"]
Nov 03 18:05:30.611958 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:30.611920    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ddbe99a-8494-4a0e-bded-a55c191297f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a83934d6-nzswt\" (UID: \"0ddbe99a-8494-4a0e-bded-a55c191297f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83934d6-nzswt"
Nov 03 18:05:30.612159 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:30.612069    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7mjn\" (UniqueName: \"kubernetes.io/projected/0ddbe99a-8494-4a0e-bded-a55c191297f3-kube-api-access-s7mjn\") pod \"maas-api-revoke-sub-a83934d6-nzswt\" (UID: \"0ddbe99a-8494-4a0e-bded-a55c191297f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83934d6-nzswt"
Nov 03 18:05:30.612159 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:30.612097    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ddbe99a-8494-4a0e-bded-a55c191297f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a83934d6-nzswt\" (UID: \"0ddbe99a-8494-4a0e-bded-a55c191297f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83934d6-nzswt"
Nov 03 18:05:30.713554 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:30.713520    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7mjn\" (UniqueName: \"kubernetes.io/projected/0ddbe99a-8494-4a0e-bded-a55c191297f3-kube-api-access-s7mjn\") pod \"maas-api-revoke-sub-a83934d6-nzswt\" (UID: \"0ddbe99a-8494-4a0e-bded-a55c191297f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83934d6-nzswt"
Nov 03 18:05:30.713554 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:30.713559    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ddbe99a-8494-4a0e-bded-a55c191297f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a83934d6-nzswt\" (UID: \"0ddbe99a-8494-4a0e-bded-a55c191297f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83934d6-nzswt"
Nov 03 18:05:30.713767 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:30.713693    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ddbe99a-8494-4a0e-bded-a55c191297f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a83934d6-nzswt\" (UID: \"0ddbe99a-8494-4a0e-bded-a55c191297f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83934d6-nzswt"
Nov 03 18:05:30.714512 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:30.714482    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ddbe99a-8494-4a0e-bded-a55c191297f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a83934d6-nzswt\" (UID: \"0ddbe99a-8494-4a0e-bded-a55c191297f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83934d6-nzswt"
Nov 03 18:05:30.716074 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:30.716053    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ddbe99a-8494-4a0e-bded-a55c191297f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a83934d6-nzswt\" (UID: \"0ddbe99a-8494-4a0e-bded-a55c191297f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83934d6-nzswt"
Nov 03 18:05:30.776661 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:30.776567    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7mjn\" (UniqueName: \"kubernetes.io/projected/0ddbe99a-8494-4a0e-bded-a55c191297f3-kube-api-access-s7mjn\") pod \"maas-api-revoke-sub-a83934d6-nzswt\" (UID: \"0ddbe99a-8494-4a0e-bded-a55c191297f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83934d6-nzswt"
Nov 03 18:05:30.846640 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:30.844208    5315 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 03 18:05:30.846640 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:30.844270    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn" podUID="ab9a094b-19e7-4bf3-a4f6-63b953d47af2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 03 18:05:30.889989 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:30.889961    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83934d6-nzswt"
Nov 03 18:05:30.892102 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:30.892076    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45755c76-x67jk" event={"ID":"2b23d8e6-0f73-4545-bbdf-e003b811018f","Type":"ContainerStarted","Data":"0f6efe0b7097b567ddb56b300f6390804f16c4f54646ef6dae15913c62956c29"}
Nov 03 18:05:31.188912 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:31.187690    5315 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 18:05:31.188912 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:31.188060    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp" podUID="3cf05bbd-6605-4cd2-aa94-85cbb46223aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 18:05:31.403520 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:31.403434    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a83934d6-nzswt"]
Nov 03 18:05:31.407527 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:05:31.407497    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ddbe99a_8494_4a0e_bded_a55c191297f3.slice/crio-71ef836034d59282d4098752f7e532444343159c7e13845b47805eb579444b82 WatchSource:0}: Error finding container 71ef836034d59282d4098752f7e532444343159c7e13845b47805eb579444b82: Status 404 returned error can't find the container with id 71ef836034d59282d4098752f7e532444343159c7e13845b47805eb579444b82
Nov 03 18:05:31.850651 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:31.848047    5315 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 03 18:05:31.850651 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:31.848108    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn" podUID="ab9a094b-19e7-4bf3-a4f6-63b953d47af2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 03 18:05:31.901160 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:31.901121    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83934d6-nzswt" event={"ID":"0ddbe99a-8494-4a0e-bded-a55c191297f3","Type":"ContainerStarted","Data":"58f9d2a1f194916214c49301560e17d6ba2d32690e581b02b2b727bf8d19ac2a"}
Nov 03 18:05:31.901542 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:31.901187    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83934d6-nzswt" event={"ID":"0ddbe99a-8494-4a0e-bded-a55c191297f3","Type":"ContainerStarted","Data":"71ef836034d59282d4098752f7e532444343159c7e13845b47805eb579444b82"}
Nov 03 18:05:31.903199 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:31.903171    5315 generic.go:356] "Generic (PLEG): container finished" podID="2b23d8e6-0f73-4545-bbdf-e003b811018f" containerID="8c99bf09bc4b707293977c3d502136baadf7401621aa4c08586d11ee48303fc8" exitCode=0
Nov 03 18:05:31.903314 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:31.903247    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45755c76-x67jk" event={"ID":"2b23d8e6-0f73-4545-bbdf-e003b811018f","Type":"ContainerDied","Data":"8c99bf09bc4b707293977c3d502136baadf7401621aa4c08586d11ee48303fc8"}
Nov 03 18:05:32.014634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:32.014521    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83934d6-nzswt" podStartSLOduration=2.014507615 podStartE2EDuration="2.014507615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:05:32.01206812 +0000 UTC m=+1171.028266189" watchObservedRunningTime="2026-11-03 18:05:32.014507615 +0000 UTC m=+1171.030705684"
Nov 03 18:05:32.076701 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:32.065660    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fe07fd3-ngtvh"]
Nov 03 18:05:32.081002 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:32.080963    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fe07fd3-ngtvh"]
Nov 03 18:05:32.181327 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:32.181292    5315 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 18:05:32.181504 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:32.181343    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp" podUID="3cf05bbd-6605-4cd2-aa94-85cbb46223aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 18:05:32.850635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:32.849889    5315 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 03 18:05:32.850635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:32.849962    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn" podUID="ab9a094b-19e7-4bf3-a4f6-63b953d47af2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 03 18:05:32.911447 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:32.911408    5315 generic.go:356] "Generic (PLEG): container finished" podID="0ddbe99a-8494-4a0e-bded-a55c191297f3" containerID="58f9d2a1f194916214c49301560e17d6ba2d32690e581b02b2b727bf8d19ac2a" exitCode=0
Nov 03 18:05:32.911864 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:32.911479    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83934d6-nzswt" event={"ID":"0ddbe99a-8494-4a0e-bded-a55c191297f3","Type":"ContainerDied","Data":"58f9d2a1f194916214c49301560e17d6ba2d32690e581b02b2b727bf8d19ac2a"}
Nov 03 18:05:33.108894 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:33.108852    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="796733e8-258e-4378-b1d2-974722376ccd" path="/var/lib/kubelet/pods/796733e8-258e-4378-b1d2-974722376ccd/volumes"
Nov 03 18:05:33.184290 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:33.183787    5315 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 18:05:33.184290 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:33.183846    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp" podUID="3cf05bbd-6605-4cd2-aa94-85cbb46223aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 18:05:33.214842 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:33.214817    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45755c76-x67jk"
Nov 03 18:05:33.316034 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:33.316001    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b23d8e6-0f73-4545-bbdf-e003b811018f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b23d8e6-0f73-4545-bbdf-e003b811018f-maas-api-xks-ca\") pod \"2b23d8e6-0f73-4545-bbdf-e003b811018f\" (UID: \"2b23d8e6-0f73-4545-bbdf-e003b811018f\") "
Nov 03 18:05:33.316247 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:33.316096    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b23d8e6-0f73-4545-bbdf-e003b811018f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b23d8e6-0f73-4545-bbdf-e003b811018f-maas-api-openshift-ca\") pod \"2b23d8e6-0f73-4545-bbdf-e003b811018f\" (UID: \"2b23d8e6-0f73-4545-bbdf-e003b811018f\") "
Nov 03 18:05:33.316319 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:33.316244    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b23d8e6-0f73-4545-bbdf-e003b811018f-kube-api-access-pslvf\" (UniqueName: \"kubernetes.io/projected/2b23d8e6-0f73-4545-bbdf-e003b811018f-kube-api-access-pslvf\") pod \"2b23d8e6-0f73-4545-bbdf-e003b811018f\" (UID: \"2b23d8e6-0f73-4545-bbdf-e003b811018f\") "
Nov 03 18:05:33.316467 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:33.316442    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b23d8e6-0f73-4545-bbdf-e003b811018f-maas-api-openshift-ca" pod "2b23d8e6-0f73-4545-bbdf-e003b811018f" (UID: "2b23d8e6-0f73-4545-bbdf-e003b811018f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:05:33.316735 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:33.316716    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b23d8e6-0f73-4545-bbdf-e003b811018f-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:05:33.318258 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:33.318232    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b23d8e6-0f73-4545-bbdf-e003b811018f-maas-api-xks-ca" pod "2b23d8e6-0f73-4545-bbdf-e003b811018f" (UID: "2b23d8e6-0f73-4545-bbdf-e003b811018f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:05:33.318543 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:33.318524    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b23d8e6-0f73-4545-bbdf-e003b811018f-kube-api-access-pslvf" pod "2b23d8e6-0f73-4545-bbdf-e003b811018f" (UID: "2b23d8e6-0f73-4545-bbdf-e003b811018f"). InnerVolumeSpecName "kube-api-access-pslvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:05:33.418101 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:33.418067    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pslvf\" (UniqueName: \"kubernetes.io/projected/2b23d8e6-0f73-4545-bbdf-e003b811018f-kube-api-access-pslvf\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:05:33.418101 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:33.418098    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b23d8e6-0f73-4545-bbdf-e003b811018f-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:05:33.841640 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:33.841581    5315 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 03 18:05:33.841825 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:33.841671    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn" podUID="ab9a094b-19e7-4bf3-a4f6-63b953d47af2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 03 18:05:33.920756 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:33.920727    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45755c76-x67jk"
Nov 03 18:05:33.921240 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:33.920760    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45755c76-x67jk" event={"ID":"2b23d8e6-0f73-4545-bbdf-e003b811018f","Type":"ContainerDied","Data":"0f6efe0b7097b567ddb56b300f6390804f16c4f54646ef6dae15913c62956c29"}
Nov 03 18:05:33.921240 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:33.920808    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f6efe0b7097b567ddb56b300f6390804f16c4f54646ef6dae15913c62956c29"
Nov 03 18:05:34.075359 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:34.075326    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83934d6-nzswt"
Nov 03 18:05:34.127367 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:34.127285    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ddbe99a-8494-4a0e-bded-a55c191297f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ddbe99a-8494-4a0e-bded-a55c191297f3-maas-api-openshift-ca\") pod \"0ddbe99a-8494-4a0e-bded-a55c191297f3\" (UID: \"0ddbe99a-8494-4a0e-bded-a55c191297f3\") "
Nov 03 18:05:34.127633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:34.127539    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ddbe99a-8494-4a0e-bded-a55c191297f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ddbe99a-8494-4a0e-bded-a55c191297f3-maas-api-xks-ca\") pod \"0ddbe99a-8494-4a0e-bded-a55c191297f3\" (UID: \"0ddbe99a-8494-4a0e-bded-a55c191297f3\") "
Nov 03 18:05:34.127633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:34.127575    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ddbe99a-8494-4a0e-bded-a55c191297f3-kube-api-access-s7mjn\" (UniqueName: \"kubernetes.io/projected/0ddbe99a-8494-4a0e-bded-a55c191297f3-kube-api-access-s7mjn\") pod \"0ddbe99a-8494-4a0e-bded-a55c191297f3\" (UID: \"0ddbe99a-8494-4a0e-bded-a55c191297f3\") "
Nov 03 18:05:34.127762 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:34.127690    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ddbe99a-8494-4a0e-bded-a55c191297f3-maas-api-openshift-ca" pod "0ddbe99a-8494-4a0e-bded-a55c191297f3" (UID: "0ddbe99a-8494-4a0e-bded-a55c191297f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:05:34.128259 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:34.128234    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ddbe99a-8494-4a0e-bded-a55c191297f3-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:05:34.130167 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:34.130144    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ddbe99a-8494-4a0e-bded-a55c191297f3-maas-api-xks-ca" pod "0ddbe99a-8494-4a0e-bded-a55c191297f3" (UID: "0ddbe99a-8494-4a0e-bded-a55c191297f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:05:34.130167 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:34.130155    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ddbe99a-8494-4a0e-bded-a55c191297f3-kube-api-access-s7mjn" pod "0ddbe99a-8494-4a0e-bded-a55c191297f3" (UID: "0ddbe99a-8494-4a0e-bded-a55c191297f3"). InnerVolumeSpecName "kube-api-access-s7mjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:05:34.181129 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:34.181089    5315 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 18:05:34.181415 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:34.181156    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp" podUID="3cf05bbd-6605-4cd2-aa94-85cbb46223aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 18:05:34.230409 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:34.230359    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ddbe99a-8494-4a0e-bded-a55c191297f3-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:05:34.230409 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:34.230406    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7mjn\" (UniqueName: \"kubernetes.io/projected/0ddbe99a-8494-4a0e-bded-a55c191297f3-kube-api-access-s7mjn\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:05:34.848813 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:34.848778    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn"
Nov 03 18:05:34.849216 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:34.849189    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn"
Nov 03 18:05:34.849923 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:34.849902    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn"
Nov 03 18:05:34.930021 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:34.929983    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83934d6-nzswt"
Nov 03 18:05:34.930021 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:34.929986    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83934d6-nzswt" event={"ID":"0ddbe99a-8494-4a0e-bded-a55c191297f3","Type":"ContainerDied","Data":"71ef836034d59282d4098752f7e532444343159c7e13845b47805eb579444b82"}
Nov 03 18:05:34.930449 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:34.930064    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71ef836034d59282d4098752f7e532444343159c7e13845b47805eb579444b82"
Nov 03 18:05:35.186508 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:35.186425    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp"
Nov 03 18:05:35.186971 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:35.186945    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp"
Nov 03 18:05:35.188017 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:35.187991    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp"
Nov 03 18:05:44.477906 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:44.477809    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2d14a24-fw975"]
Nov 03 18:05:44.478514 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:44.478493    5315 cpu_manager.go:414] "containerMap: removing container" podUID="0ddbe99a-8494-4a0e-bded-a55c191297f3" containerName="revoke-keys"
Nov 03 18:05:44.478514 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:44.478514    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ddbe99a-8494-4a0e-bded-a55c191297f3" containerName="revoke-keys"
Nov 03 18:05:44.478700 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:44.478560    5315 cpu_manager.go:414] "containerMap: removing container" podUID="2b23d8e6-0f73-4545-bbdf-e003b811018f" containerName="revoke-keys"
Nov 03 18:05:44.478700 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:44.478569    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b23d8e6-0f73-4545-bbdf-e003b811018f" containerName="revoke-keys"
Nov 03 18:05:44.478783 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:44.478722    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ddbe99a-8494-4a0e-bded-a55c191297f3" containerName="revoke-keys"
Nov 03 18:05:44.478783 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:44.478739    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b23d8e6-0f73-4545-bbdf-e003b811018f" containerName="revoke-keys"
Nov 03 18:05:44.483354 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:44.483330    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2d14a24-fw975"
Nov 03 18:05:44.531957 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:44.531917    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:05:44.532133 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:44.531967    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:05:44.532133 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:44.532000    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:05:44.590639 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:44.590251    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a91be802-1c0f-4e1a-8780-261497c1b71d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2d14a24-fw975\" (UID: \"a91be802-1c0f-4e1a-8780-261497c1b71d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2d14a24-fw975"
Nov 03 18:05:44.590639 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:44.590332    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95cs6\" (UniqueName: \"kubernetes.io/projected/a91be802-1c0f-4e1a-8780-261497c1b71d-kube-api-access-95cs6\") pod \"maas-api-revoke-sub-e2d14a24-fw975\" (UID: \"a91be802-1c0f-4e1a-8780-261497c1b71d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2d14a24-fw975"
Nov 03 18:05:44.590639 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:44.590508    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a91be802-1c0f-4e1a-8780-261497c1b71d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2d14a24-fw975\" (UID: \"a91be802-1c0f-4e1a-8780-261497c1b71d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2d14a24-fw975"
Nov 03 18:05:44.590947 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:44.590903    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:05:44.600676 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:44.596129    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2d14a24-fw975"]
Nov 03 18:05:44.692184 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:44.692147    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95cs6\" (UniqueName: \"kubernetes.io/projected/a91be802-1c0f-4e1a-8780-261497c1b71d-kube-api-access-95cs6\") pod \"maas-api-revoke-sub-e2d14a24-fw975\" (UID: \"a91be802-1c0f-4e1a-8780-261497c1b71d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2d14a24-fw975"
Nov 03 18:05:44.692332 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:44.692263    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a91be802-1c0f-4e1a-8780-261497c1b71d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2d14a24-fw975\" (UID: \"a91be802-1c0f-4e1a-8780-261497c1b71d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2d14a24-fw975"
Nov 03 18:05:44.692396 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:44.692344    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a91be802-1c0f-4e1a-8780-261497c1b71d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2d14a24-fw975\" (UID: \"a91be802-1c0f-4e1a-8780-261497c1b71d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2d14a24-fw975"
Nov 03 18:05:44.693034 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:44.693006    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a91be802-1c0f-4e1a-8780-261497c1b71d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2d14a24-fw975\" (UID: \"a91be802-1c0f-4e1a-8780-261497c1b71d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2d14a24-fw975"
Nov 03 18:05:44.694677 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:44.694657    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a91be802-1c0f-4e1a-8780-261497c1b71d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2d14a24-fw975\" (UID: \"a91be802-1c0f-4e1a-8780-261497c1b71d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2d14a24-fw975"
Nov 03 18:05:44.749991 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:44.749914    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95cs6\" (UniqueName: \"kubernetes.io/projected/a91be802-1c0f-4e1a-8780-261497c1b71d-kube-api-access-95cs6\") pod \"maas-api-revoke-sub-e2d14a24-fw975\" (UID: \"a91be802-1c0f-4e1a-8780-261497c1b71d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2d14a24-fw975"
Nov 03 18:05:44.797823 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:44.797782    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2d14a24-fw975"
Nov 03 18:05:45.172392 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:45.172346    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2d14a24-fw975"]
Nov 03 18:05:45.177836 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:05:45.177803    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda91be802_1c0f_4e1a_8780_261497c1b71d.slice/crio-3627d5f3d7888d01400b820fd5b545dc11326dd3bfd71e9e16148990f62ea05d WatchSource:0}: Error finding container 3627d5f3d7888d01400b820fd5b545dc11326dd3bfd71e9e16148990f62ea05d: Status 404 returned error can't find the container with id 3627d5f3d7888d01400b820fd5b545dc11326dd3bfd71e9e16148990f62ea05d
Nov 03 18:05:46.009457 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:46.009418    5315 generic.go:356] "Generic (PLEG): container finished" podID="a91be802-1c0f-4e1a-8780-261497c1b71d" containerID="ee9d5c2dbdc7922a7f44703613015c2efa1357033170842c6a05c431280375da" exitCode=0
Nov 03 18:05:46.009942 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:46.009494    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2d14a24-fw975" event={"ID":"a91be802-1c0f-4e1a-8780-261497c1b71d","Type":"ContainerDied","Data":"ee9d5c2dbdc7922a7f44703613015c2efa1357033170842c6a05c431280375da"}
Nov 03 18:05:46.009942 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:46.009557    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2d14a24-fw975" event={"ID":"a91be802-1c0f-4e1a-8780-261497c1b71d","Type":"ContainerStarted","Data":"3627d5f3d7888d01400b820fd5b545dc11326dd3bfd71e9e16148990f62ea05d"}
Nov 03 18:05:47.971634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:47.957174    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7" podUID="69760dba-faf9-4c08-895c-ad70fe95fd49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/ready\": context deadline exceeded"
Nov 03 18:05:47.995634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:47.985693    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c" podUID="8ac59539-8ac1-466e-8d1b-b2be4c42c708" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:05:48.015635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:48.009738    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7" podUID="33725a51-6459-4f15-a866-e9dee41a9732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:05:48.015635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:48.013836    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2d14a24-fw975"
Nov 03 18:05:48.114839 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:48.114731    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s" podUID="155e4b39-1877-4bd9-8541-5d6bd3aa0a8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:05:48.144644 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:48.142554    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2d14a24-fw975" event={"ID":"a91be802-1c0f-4e1a-8780-261497c1b71d","Type":"ContainerDied","Data":"3627d5f3d7888d01400b820fd5b545dc11326dd3bfd71e9e16148990f62ea05d"}
Nov 03 18:05:48.144965 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:48.144939    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2d14a24-fw975"
Nov 03 18:05:48.153634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:48.151671    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3627d5f3d7888d01400b820fd5b545dc11326dd3bfd71e9e16148990f62ea05d"
Nov 03 18:05:48.157735 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:48.157694    5315 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv" podUID="f6d706cb-3aa5-419c-8f6c-29347bb788b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:05:48.165173 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:48.165141    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a91be802-1c0f-4e1a-8780-261497c1b71d-kube-api-access-95cs6\" (UniqueName: \"kubernetes.io/projected/a91be802-1c0f-4e1a-8780-261497c1b71d-kube-api-access-95cs6\") pod \"a91be802-1c0f-4e1a-8780-261497c1b71d\" (UID: \"a91be802-1c0f-4e1a-8780-261497c1b71d\") "
Nov 03 18:05:48.165462 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:48.165437    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a91be802-1c0f-4e1a-8780-261497c1b71d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a91be802-1c0f-4e1a-8780-261497c1b71d-maas-api-openshift-ca\") pod \"a91be802-1c0f-4e1a-8780-261497c1b71d\" (UID: \"a91be802-1c0f-4e1a-8780-261497c1b71d\") "
Nov 03 18:05:48.165599 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:48.165581    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a91be802-1c0f-4e1a-8780-261497c1b71d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a91be802-1c0f-4e1a-8780-261497c1b71d-maas-api-xks-ca\") pod \"a91be802-1c0f-4e1a-8780-261497c1b71d\" (UID: \"a91be802-1c0f-4e1a-8780-261497c1b71d\") "
Nov 03 18:05:48.185147 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:48.185104    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l" podUID="114dcf54-e4eb-4294-a77b-604fe6d57d48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:05:48.200900 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:48.200857    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a91be802-1c0f-4e1a-8780-261497c1b71d-maas-api-openshift-ca" pod "a91be802-1c0f-4e1a-8780-261497c1b71d" (UID: "a91be802-1c0f-4e1a-8780-261497c1b71d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:05:48.201567 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:48.201536    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a91be802-1c0f-4e1a-8780-261497c1b71d-maas-api-xks-ca" pod "a91be802-1c0f-4e1a-8780-261497c1b71d" (UID: "a91be802-1c0f-4e1a-8780-261497c1b71d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:05:48.257777 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:48.257722    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt" podUID="7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Nov 03 18:05:48.271047 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:48.271003    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a91be802-1c0f-4e1a-8780-261497c1b71d-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:05:48.276842 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:48.276812    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a91be802-1c0f-4e1a-8780-261497c1b71d-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:05:48.284778 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:48.284730    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a91be802-1c0f-4e1a-8780-261497c1b71d-kube-api-access-95cs6" pod "a91be802-1c0f-4e1a-8780-261497c1b71d" (UID: "a91be802-1c0f-4e1a-8780-261497c1b71d"). InnerVolumeSpecName "kube-api-access-95cs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:05:48.378400 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:48.378316    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95cs6\" (UniqueName: \"kubernetes.io/projected/a91be802-1c0f-4e1a-8780-261497c1b71d-kube-api-access-95cs6\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:05:48.952698 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:48.952663    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fdfba9b-x4wnb"]
Nov 03 18:05:48.953554 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:48.953533    5315 cpu_manager.go:414] "containerMap: removing container" podUID="a91be802-1c0f-4e1a-8780-261497c1b71d" containerName="revoke-keys"
Nov 03 18:05:48.953554 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:48.953553    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a91be802-1c0f-4e1a-8780-261497c1b71d" containerName="revoke-keys"
Nov 03 18:05:48.953730 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:48.953720    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="a91be802-1c0f-4e1a-8780-261497c1b71d" containerName="revoke-keys"
Nov 03 18:05:48.957648 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:48.957629    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fdfba9b-x4wnb"
Nov 03 18:05:49.006282 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:49.006246    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:05:49.006282 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:49.006266    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:05:49.006831 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:49.006386    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:05:49.006831 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:49.006408    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:05:49.060458 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:49.060427    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fdfba9b-x4wnb"]
Nov 03 18:05:49.086961 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:49.086925    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd07730f-0cf5-4a0a-b21e-e9b96dd7942a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0fdfba9b-x4wnb\" (UID: \"fd07730f-0cf5-4a0a-b21e-e9b96dd7942a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fdfba9b-x4wnb"
Nov 03 18:05:49.087174 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:49.086975    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8wwc\" (UniqueName: \"kubernetes.io/projected/fd07730f-0cf5-4a0a-b21e-e9b96dd7942a-kube-api-access-r8wwc\") pod \"maas-api-revoke-sub-0fdfba9b-x4wnb\" (UID: \"fd07730f-0cf5-4a0a-b21e-e9b96dd7942a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fdfba9b-x4wnb"
Nov 03 18:05:49.087247 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:49.087205    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd07730f-0cf5-4a0a-b21e-e9b96dd7942a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0fdfba9b-x4wnb\" (UID: \"fd07730f-0cf5-4a0a-b21e-e9b96dd7942a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fdfba9b-x4wnb"
Nov 03 18:05:49.096471 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:49.096397    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7995a32-gl2x5"]
Nov 03 18:05:49.100421 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:49.100401    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7995a32-gl2x5"
Nov 03 18:05:49.217820 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:49.211086    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd07730f-0cf5-4a0a-b21e-e9b96dd7942a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0fdfba9b-x4wnb\" (UID: \"fd07730f-0cf5-4a0a-b21e-e9b96dd7942a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fdfba9b-x4wnb"
Nov 03 18:05:49.218185 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:49.218152    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6fhr\" (UniqueName: \"kubernetes.io/projected/ca2b4231-a18a-4290-afee-852f0db90ea4-kube-api-access-h6fhr\") pod \"maas-api-revoke-sub-d7995a32-gl2x5\" (UID: \"ca2b4231-a18a-4290-afee-852f0db90ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7995a32-gl2x5"
Nov 03 18:05:49.218441 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:49.218420    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca2b4231-a18a-4290-afee-852f0db90ea4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7995a32-gl2x5\" (UID: \"ca2b4231-a18a-4290-afee-852f0db90ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7995a32-gl2x5"
Nov 03 18:05:49.218578 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:49.218560    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca2b4231-a18a-4290-afee-852f0db90ea4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7995a32-gl2x5\" (UID: \"ca2b4231-a18a-4290-afee-852f0db90ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7995a32-gl2x5"
Nov 03 18:05:49.218818 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:49.218796    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd07730f-0cf5-4a0a-b21e-e9b96dd7942a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0fdfba9b-x4wnb\" (UID: \"fd07730f-0cf5-4a0a-b21e-e9b96dd7942a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fdfba9b-x4wnb"
Nov 03 18:05:49.218926 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:49.217808    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd07730f-0cf5-4a0a-b21e-e9b96dd7942a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0fdfba9b-x4wnb\" (UID: \"fd07730f-0cf5-4a0a-b21e-e9b96dd7942a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fdfba9b-x4wnb"
Nov 03 18:05:49.219021 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:49.218975    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8wwc\" (UniqueName: \"kubernetes.io/projected/fd07730f-0cf5-4a0a-b21e-e9b96dd7942a-kube-api-access-r8wwc\") pod \"maas-api-revoke-sub-0fdfba9b-x4wnb\" (UID: \"fd07730f-0cf5-4a0a-b21e-e9b96dd7942a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fdfba9b-x4wnb"
Nov 03 18:05:49.227954 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:49.227913    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd07730f-0cf5-4a0a-b21e-e9b96dd7942a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0fdfba9b-x4wnb\" (UID: \"fd07730f-0cf5-4a0a-b21e-e9b96dd7942a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fdfba9b-x4wnb"
Nov 03 18:05:49.228185 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:49.228168    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7995a32-gl2x5"]
Nov 03 18:05:49.276444 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:49.276405    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8wwc\" (UniqueName: \"kubernetes.io/projected/fd07730f-0cf5-4a0a-b21e-e9b96dd7942a-kube-api-access-r8wwc\") pod \"maas-api-revoke-sub-0fdfba9b-x4wnb\" (UID: \"fd07730f-0cf5-4a0a-b21e-e9b96dd7942a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fdfba9b-x4wnb"
Nov 03 18:05:49.320828 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:49.320789    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6fhr\" (UniqueName: \"kubernetes.io/projected/ca2b4231-a18a-4290-afee-852f0db90ea4-kube-api-access-h6fhr\") pod \"maas-api-revoke-sub-d7995a32-gl2x5\" (UID: \"ca2b4231-a18a-4290-afee-852f0db90ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7995a32-gl2x5"
Nov 03 18:05:49.321021 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:49.320889    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca2b4231-a18a-4290-afee-852f0db90ea4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7995a32-gl2x5\" (UID: \"ca2b4231-a18a-4290-afee-852f0db90ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7995a32-gl2x5"
Nov 03 18:05:49.321021 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:49.320921    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca2b4231-a18a-4290-afee-852f0db90ea4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7995a32-gl2x5\" (UID: \"ca2b4231-a18a-4290-afee-852f0db90ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7995a32-gl2x5"
Nov 03 18:05:49.321691 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:49.321665    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca2b4231-a18a-4290-afee-852f0db90ea4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7995a32-gl2x5\" (UID: \"ca2b4231-a18a-4290-afee-852f0db90ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7995a32-gl2x5"
Nov 03 18:05:49.323183 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:49.323156    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca2b4231-a18a-4290-afee-852f0db90ea4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7995a32-gl2x5\" (UID: \"ca2b4231-a18a-4290-afee-852f0db90ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7995a32-gl2x5"
Nov 03 18:05:49.378737 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:49.378659    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6fhr\" (UniqueName: \"kubernetes.io/projected/ca2b4231-a18a-4290-afee-852f0db90ea4-kube-api-access-h6fhr\") pod \"maas-api-revoke-sub-d7995a32-gl2x5\" (UID: \"ca2b4231-a18a-4290-afee-852f0db90ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7995a32-gl2x5"
Nov 03 18:05:49.413591 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:49.413557    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7995a32-gl2x5"
Nov 03 18:05:49.572788 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:49.572749    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fdfba9b-x4wnb"
Nov 03 18:05:49.791326 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:49.791285    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7995a32-gl2x5"]
Nov 03 18:05:49.797222 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:05:49.797189    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca2b4231_a18a_4290_afee_852f0db90ea4.slice/crio-9ba9b29557b4dd700cfa33caf30a3ea9c6e27a0e46696220e07290c2dde5373d WatchSource:0}: Error finding container 9ba9b29557b4dd700cfa33caf30a3ea9c6e27a0e46696220e07290c2dde5373d: Status 404 returned error can't find the container with id 9ba9b29557b4dd700cfa33caf30a3ea9c6e27a0e46696220e07290c2dde5373d
Nov 03 18:05:49.946247 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:49.946150    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fdfba9b-x4wnb"]
Nov 03 18:05:50.026504 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:05:50.026468    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd07730f_0cf5_4a0a_b21e_e9b96dd7942a.slice/crio-b9b4cf7ea2f9d4fcc97abc3c5f3ab0ef0b7bd009b3a40036d048a5beb08f57aa WatchSource:0}: Error finding container b9b4cf7ea2f9d4fcc97abc3c5f3ab0ef0b7bd009b3a40036d048a5beb08f57aa: Status 404 returned error can't find the container with id b9b4cf7ea2f9d4fcc97abc3c5f3ab0ef0b7bd009b3a40036d048a5beb08f57aa
Nov 03 18:05:50.065728 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:50.065689    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f298b25a-pjz76"]
Nov 03 18:05:50.069469 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:50.069443    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f298b25a-pjz76"]
Nov 03 18:05:50.159571 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:50.159538    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fdfba9b-x4wnb" event={"ID":"fd07730f-0cf5-4a0a-b21e-e9b96dd7942a","Type":"ContainerStarted","Data":"b9b4cf7ea2f9d4fcc97abc3c5f3ab0ef0b7bd009b3a40036d048a5beb08f57aa"}
Nov 03 18:05:50.160906 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:50.160880    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7995a32-gl2x5" event={"ID":"ca2b4231-a18a-4290-afee-852f0db90ea4","Type":"ContainerStarted","Data":"9ba9b29557b4dd700cfa33caf30a3ea9c6e27a0e46696220e07290c2dde5373d"}
Nov 03 18:05:50.936275 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:50.936239    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37d7625f-rqgpz"]
Nov 03 18:05:50.940753 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:50.940729    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d7625f-rqgpz"
Nov 03 18:05:50.945280 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:50.945242    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e598e385-1a33-43c4-846b-8cc4e29f138e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37d7625f-rqgpz\" (UID: \"e598e385-1a33-43c4-846b-8cc4e29f138e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d7625f-rqgpz"
Nov 03 18:05:50.945425 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:50.945412    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e598e385-1a33-43c4-846b-8cc4e29f138e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37d7625f-rqgpz\" (UID: \"e598e385-1a33-43c4-846b-8cc4e29f138e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d7625f-rqgpz"
Nov 03 18:05:50.945471 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:50.945462    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnzgl\" (UniqueName: \"kubernetes.io/projected/e598e385-1a33-43c4-846b-8cc4e29f138e-kube-api-access-cnzgl\") pod \"maas-api-revoke-sub-37d7625f-rqgpz\" (UID: \"e598e385-1a33-43c4-846b-8cc4e29f138e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d7625f-rqgpz"
Nov 03 18:05:50.999538 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:50.999506    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37d7625f-rqgpz"]
Nov 03 18:05:51.047038 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:51.047002    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e598e385-1a33-43c4-846b-8cc4e29f138e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37d7625f-rqgpz\" (UID: \"e598e385-1a33-43c4-846b-8cc4e29f138e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d7625f-rqgpz"
Nov 03 18:05:51.047432 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:51.047078    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnzgl\" (UniqueName: \"kubernetes.io/projected/e598e385-1a33-43c4-846b-8cc4e29f138e-kube-api-access-cnzgl\") pod \"maas-api-revoke-sub-37d7625f-rqgpz\" (UID: \"e598e385-1a33-43c4-846b-8cc4e29f138e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d7625f-rqgpz"
Nov 03 18:05:51.047432 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:51.047204    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e598e385-1a33-43c4-846b-8cc4e29f138e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37d7625f-rqgpz\" (UID: \"e598e385-1a33-43c4-846b-8cc4e29f138e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d7625f-rqgpz"
Nov 03 18:05:51.047848 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:51.047817    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e598e385-1a33-43c4-846b-8cc4e29f138e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37d7625f-rqgpz\" (UID: \"e598e385-1a33-43c4-846b-8cc4e29f138e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d7625f-rqgpz"
Nov 03 18:05:51.049713 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:51.049689    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e598e385-1a33-43c4-846b-8cc4e29f138e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37d7625f-rqgpz\" (UID: \"e598e385-1a33-43c4-846b-8cc4e29f138e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d7625f-rqgpz"
Nov 03 18:05:51.105357 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:51.105318    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnzgl\" (UniqueName: \"kubernetes.io/projected/e598e385-1a33-43c4-846b-8cc4e29f138e-kube-api-access-cnzgl\") pod \"maas-api-revoke-sub-37d7625f-rqgpz\" (UID: \"e598e385-1a33-43c4-846b-8cc4e29f138e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d7625f-rqgpz"
Nov 03 18:05:51.112964 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:51.112928    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e8bb82a-0c44-491b-a255-ee8e5c925ab2" path="/var/lib/kubelet/pods/7e8bb82a-0c44-491b-a255-ee8e5c925ab2/volumes"
Nov 03 18:05:51.169756 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:51.169719    5315 generic.go:356] "Generic (PLEG): container finished" podID="fd07730f-0cf5-4a0a-b21e-e9b96dd7942a" containerID="3197f6e1c35cf3b2bd620b9cda58e8647a0eeab462449216feeba603c2379491" exitCode=0
Nov 03 18:05:51.169926 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:51.169802    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fdfba9b-x4wnb" event={"ID":"fd07730f-0cf5-4a0a-b21e-e9b96dd7942a","Type":"ContainerDied","Data":"3197f6e1c35cf3b2bd620b9cda58e8647a0eeab462449216feeba603c2379491"}
Nov 03 18:05:51.171417 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:51.171389    5315 generic.go:356] "Generic (PLEG): container finished" podID="ca2b4231-a18a-4290-afee-852f0db90ea4" containerID="8a01bbaae0ee125f7fbd5adee0a7922fc4bf93d9d7a4a0462e8742f2898cb9ad" exitCode=0
Nov 03 18:05:51.171559 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:51.171476    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7995a32-gl2x5" event={"ID":"ca2b4231-a18a-4290-afee-852f0db90ea4","Type":"ContainerDied","Data":"8a01bbaae0ee125f7fbd5adee0a7922fc4bf93d9d7a4a0462e8742f2898cb9ad"}
Nov 03 18:05:51.254960 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:51.254920    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d7625f-rqgpz"
Nov 03 18:05:51.630391 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:51.630302    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37d7625f-rqgpz"]
Nov 03 18:05:51.635013 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:05:51.634978    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode598e385_1a33_43c4_846b_8cc4e29f138e.slice/crio-505b2b3265a65f7976b8791566b14f1fff8e6062e2dbd7e54a2340ffb68777e7 WatchSource:0}: Error finding container 505b2b3265a65f7976b8791566b14f1fff8e6062e2dbd7e54a2340ffb68777e7: Status 404 returned error can't find the container with id 505b2b3265a65f7976b8791566b14f1fff8e6062e2dbd7e54a2340ffb68777e7
Nov 03 18:05:52.180427 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:52.180384    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d7625f-rqgpz" event={"ID":"e598e385-1a33-43c4-846b-8cc4e29f138e","Type":"ContainerStarted","Data":"306a5dc9f82a9f761f7d1af179ba9b920a5e34a83fbb2d2f7addc55c458651dc"}
Nov 03 18:05:52.180892 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:52.180455    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d7625f-rqgpz" event={"ID":"e598e385-1a33-43c4-846b-8cc4e29f138e","Type":"ContainerStarted","Data":"505b2b3265a65f7976b8791566b14f1fff8e6062e2dbd7e54a2340ffb68777e7"}
Nov 03 18:05:52.291758 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:52.291707    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d7625f-rqgpz" podStartSLOduration=2.291692903 podStartE2EDuration="2.291692903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:05:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:05:52.288439539 +0000 UTC m=+1191.304637621" watchObservedRunningTime="2026-11-03 18:05:52.291692903 +0000 UTC m=+1191.307891000"
Nov 03 18:05:52.347097 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:52.347053    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7995a32-gl2x5"
Nov 03 18:05:52.370027 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:52.369989    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca2b4231-a18a-4290-afee-852f0db90ea4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca2b4231-a18a-4290-afee-852f0db90ea4-maas-api-openshift-ca\") pod \"ca2b4231-a18a-4290-afee-852f0db90ea4\" (UID: \"ca2b4231-a18a-4290-afee-852f0db90ea4\") "
Nov 03 18:05:52.370165 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:52.370039    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca2b4231-a18a-4290-afee-852f0db90ea4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca2b4231-a18a-4290-afee-852f0db90ea4-maas-api-xks-ca\") pod \"ca2b4231-a18a-4290-afee-852f0db90ea4\" (UID: \"ca2b4231-a18a-4290-afee-852f0db90ea4\") "
Nov 03 18:05:52.370165 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:52.370077    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca2b4231-a18a-4290-afee-852f0db90ea4-kube-api-access-h6fhr\" (UniqueName: \"kubernetes.io/projected/ca2b4231-a18a-4290-afee-852f0db90ea4-kube-api-access-h6fhr\") pod \"ca2b4231-a18a-4290-afee-852f0db90ea4\" (UID: \"ca2b4231-a18a-4290-afee-852f0db90ea4\") "
Nov 03 18:05:52.370367 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:52.370331    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca2b4231-a18a-4290-afee-852f0db90ea4-maas-api-openshift-ca" pod "ca2b4231-a18a-4290-afee-852f0db90ea4" (UID: "ca2b4231-a18a-4290-afee-852f0db90ea4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:05:52.370720 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:52.370688    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca2b4231-a18a-4290-afee-852f0db90ea4-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:05:52.372205 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:52.372179    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca2b4231-a18a-4290-afee-852f0db90ea4-maas-api-xks-ca" pod "ca2b4231-a18a-4290-afee-852f0db90ea4" (UID: "ca2b4231-a18a-4290-afee-852f0db90ea4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:05:52.372282 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:52.372261    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca2b4231-a18a-4290-afee-852f0db90ea4-kube-api-access-h6fhr" pod "ca2b4231-a18a-4290-afee-852f0db90ea4" (UID: "ca2b4231-a18a-4290-afee-852f0db90ea4"). InnerVolumeSpecName "kube-api-access-h6fhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:05:52.394377 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:52.394329    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fdfba9b-x4wnb"
Nov 03 18:05:52.472152 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:52.472117    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd07730f-0cf5-4a0a-b21e-e9b96dd7942a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd07730f-0cf5-4a0a-b21e-e9b96dd7942a-maas-api-openshift-ca\") pod \"fd07730f-0cf5-4a0a-b21e-e9b96dd7942a\" (UID: \"fd07730f-0cf5-4a0a-b21e-e9b96dd7942a\") "
Nov 03 18:05:52.472320 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:52.472200    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd07730f-0cf5-4a0a-b21e-e9b96dd7942a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd07730f-0cf5-4a0a-b21e-e9b96dd7942a-maas-api-xks-ca\") pod \"fd07730f-0cf5-4a0a-b21e-e9b96dd7942a\" (UID: \"fd07730f-0cf5-4a0a-b21e-e9b96dd7942a\") "
Nov 03 18:05:52.472320 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:52.472237    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd07730f-0cf5-4a0a-b21e-e9b96dd7942a-kube-api-access-r8wwc\" (UniqueName: \"kubernetes.io/projected/fd07730f-0cf5-4a0a-b21e-e9b96dd7942a-kube-api-access-r8wwc\") pod \"fd07730f-0cf5-4a0a-b21e-e9b96dd7942a\" (UID: \"fd07730f-0cf5-4a0a-b21e-e9b96dd7942a\") "
Nov 03 18:05:52.472596 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:52.472565    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd07730f-0cf5-4a0a-b21e-e9b96dd7942a-maas-api-openshift-ca" pod "fd07730f-0cf5-4a0a-b21e-e9b96dd7942a" (UID: "fd07730f-0cf5-4a0a-b21e-e9b96dd7942a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:05:52.472891 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:52.472863    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd07730f-0cf5-4a0a-b21e-e9b96dd7942a-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:05:52.473006 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:52.472895    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca2b4231-a18a-4290-afee-852f0db90ea4-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:05:52.473006 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:52.472913    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6fhr\" (UniqueName: \"kubernetes.io/projected/ca2b4231-a18a-4290-afee-852f0db90ea4-kube-api-access-h6fhr\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:05:52.474407 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:52.474381    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd07730f-0cf5-4a0a-b21e-e9b96dd7942a-maas-api-xks-ca" pod "fd07730f-0cf5-4a0a-b21e-e9b96dd7942a" (UID: "fd07730f-0cf5-4a0a-b21e-e9b96dd7942a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:05:52.474881 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:52.474856    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd07730f-0cf5-4a0a-b21e-e9b96dd7942a-kube-api-access-r8wwc" pod "fd07730f-0cf5-4a0a-b21e-e9b96dd7942a" (UID: "fd07730f-0cf5-4a0a-b21e-e9b96dd7942a"). InnerVolumeSpecName "kube-api-access-r8wwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:05:52.574020 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:52.573973    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd07730f-0cf5-4a0a-b21e-e9b96dd7942a-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:05:52.574020 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:52.574021    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8wwc\" (UniqueName: \"kubernetes.io/projected/fd07730f-0cf5-4a0a-b21e-e9b96dd7942a-kube-api-access-r8wwc\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:05:53.073315 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:53.073269    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b75ece4b-f94kw"]
Nov 03 18:05:53.073962 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:53.073940    5315 cpu_manager.go:414] "containerMap: removing container" podUID="fd07730f-0cf5-4a0a-b21e-e9b96dd7942a" containerName="revoke-keys"
Nov 03 18:05:53.073962 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:53.073961    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd07730f-0cf5-4a0a-b21e-e9b96dd7942a" containerName="revoke-keys"
Nov 03 18:05:53.074174 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:53.073993    5315 cpu_manager.go:414] "containerMap: removing container" podUID="ca2b4231-a18a-4290-afee-852f0db90ea4" containerName="revoke-keys"
Nov 03 18:05:53.074174 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:53.074007    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca2b4231-a18a-4290-afee-852f0db90ea4" containerName="revoke-keys"
Nov 03 18:05:53.074174 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:53.074132    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca2b4231-a18a-4290-afee-852f0db90ea4" containerName="revoke-keys"
Nov 03 18:05:53.074174 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:53.074156    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd07730f-0cf5-4a0a-b21e-e9b96dd7942a" containerName="revoke-keys"
Nov 03 18:05:53.077815 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:53.077792    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75ece4b-f94kw"
Nov 03 18:05:53.110163 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:53.110127    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2610a6e1-9nwl5"]
Nov 03 18:05:53.118486 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:53.118433    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2610a6e1-9nwl5"
Nov 03 18:05:53.140444 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:53.140405    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b75ece4b-f94kw"]
Nov 03 18:05:53.181039 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:53.180921    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl5m9\" (UniqueName: \"kubernetes.io/projected/510cff64-6154-42a0-8657-5a99aa556133-kube-api-access-wl5m9\") pod \"maas-api-revoke-sub-b75ece4b-f94kw\" (UID: \"510cff64-6154-42a0-8657-5a99aa556133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75ece4b-f94kw"
Nov 03 18:05:53.181039 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:53.181004    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/510cff64-6154-42a0-8657-5a99aa556133-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b75ece4b-f94kw\" (UID: \"510cff64-6154-42a0-8657-5a99aa556133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75ece4b-f94kw"
Nov 03 18:05:53.181643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:53.181097    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d33f7de6-f94f-4d92-8832-5bddc50358e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2610a6e1-9nwl5\" (UID: \"d33f7de6-f94f-4d92-8832-5bddc50358e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2610a6e1-9nwl5"
Nov 03 18:05:53.181643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:53.181121    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-275g7\" (UniqueName: \"kubernetes.io/projected/d33f7de6-f94f-4d92-8832-5bddc50358e4-kube-api-access-275g7\") pod \"maas-api-revoke-sub-2610a6e1-9nwl5\" (UID: \"d33f7de6-f94f-4d92-8832-5bddc50358e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2610a6e1-9nwl5"
Nov 03 18:05:53.181643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:53.181289    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/510cff64-6154-42a0-8657-5a99aa556133-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b75ece4b-f94kw\" (UID: \"510cff64-6154-42a0-8657-5a99aa556133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75ece4b-f94kw"
Nov 03 18:05:53.181643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:53.181423    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d33f7de6-f94f-4d92-8832-5bddc50358e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2610a6e1-9nwl5\" (UID: \"d33f7de6-f94f-4d92-8832-5bddc50358e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2610a6e1-9nwl5"
Nov 03 18:05:53.185159 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:53.185133    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2610a6e1-9nwl5"]
Nov 03 18:05:53.192283 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:53.192258    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7995a32-gl2x5"
Nov 03 18:05:53.192553 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:53.192487    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7995a32-gl2x5" event={"ID":"ca2b4231-a18a-4290-afee-852f0db90ea4","Type":"ContainerDied","Data":"9ba9b29557b4dd700cfa33caf30a3ea9c6e27a0e46696220e07290c2dde5373d"}
Nov 03 18:05:53.192885 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:53.192860    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ba9b29557b4dd700cfa33caf30a3ea9c6e27a0e46696220e07290c2dde5373d"
Nov 03 18:05:53.197893 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:53.197853    5315 generic.go:356] "Generic (PLEG): container finished" podID="e598e385-1a33-43c4-846b-8cc4e29f138e" containerID="306a5dc9f82a9f761f7d1af179ba9b920a5e34a83fbb2d2f7addc55c458651dc" exitCode=0
Nov 03 18:05:53.198380 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:53.198030    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d7625f-rqgpz" event={"ID":"e598e385-1a33-43c4-846b-8cc4e29f138e","Type":"ContainerDied","Data":"306a5dc9f82a9f761f7d1af179ba9b920a5e34a83fbb2d2f7addc55c458651dc"}
Nov 03 18:05:53.199777 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:53.199756    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fdfba9b-x4wnb" event={"ID":"fd07730f-0cf5-4a0a-b21e-e9b96dd7942a","Type":"ContainerDied","Data":"b9b4cf7ea2f9d4fcc97abc3c5f3ab0ef0b7bd009b3a40036d048a5beb08f57aa"}
Nov 03 18:05:53.199868 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:53.199792    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fdfba9b-x4wnb"
Nov 03 18:05:53.199978 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:53.199795    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9b4cf7ea2f9d4fcc97abc3c5f3ab0ef0b7bd009b3a40036d048a5beb08f57aa"
Nov 03 18:05:53.282873 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:53.282842    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d33f7de6-f94f-4d92-8832-5bddc50358e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2610a6e1-9nwl5\" (UID: \"d33f7de6-f94f-4d92-8832-5bddc50358e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2610a6e1-9nwl5"
Nov 03 18:05:53.283079 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:53.282897    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl5m9\" (UniqueName: \"kubernetes.io/projected/510cff64-6154-42a0-8657-5a99aa556133-kube-api-access-wl5m9\") pod \"maas-api-revoke-sub-b75ece4b-f94kw\" (UID: \"510cff64-6154-42a0-8657-5a99aa556133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75ece4b-f94kw"
Nov 03 18:05:53.283079 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:53.282933    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/510cff64-6154-42a0-8657-5a99aa556133-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b75ece4b-f94kw\" (UID: \"510cff64-6154-42a0-8657-5a99aa556133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75ece4b-f94kw"
Nov 03 18:05:53.283079 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:53.282959    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d33f7de6-f94f-4d92-8832-5bddc50358e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2610a6e1-9nwl5\" (UID: \"d33f7de6-f94f-4d92-8832-5bddc50358e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2610a6e1-9nwl5"
Nov 03 18:05:53.283079 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:53.282974    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-275g7\" (UniqueName: \"kubernetes.io/projected/d33f7de6-f94f-4d92-8832-5bddc50358e4-kube-api-access-275g7\") pod \"maas-api-revoke-sub-2610a6e1-9nwl5\" (UID: \"d33f7de6-f94f-4d92-8832-5bddc50358e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2610a6e1-9nwl5"
Nov 03 18:05:53.283079 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:53.283032    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/510cff64-6154-42a0-8657-5a99aa556133-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b75ece4b-f94kw\" (UID: \"510cff64-6154-42a0-8657-5a99aa556133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75ece4b-f94kw"
Nov 03 18:05:53.283638 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:53.283593    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d33f7de6-f94f-4d92-8832-5bddc50358e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2610a6e1-9nwl5\" (UID: \"d33f7de6-f94f-4d92-8832-5bddc50358e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2610a6e1-9nwl5"
Nov 03 18:05:53.283770 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:53.283744    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/510cff64-6154-42a0-8657-5a99aa556133-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b75ece4b-f94kw\" (UID: \"510cff64-6154-42a0-8657-5a99aa556133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75ece4b-f94kw"
Nov 03 18:05:53.285314 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:53.285264    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/510cff64-6154-42a0-8657-5a99aa556133-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b75ece4b-f94kw\" (UID: \"510cff64-6154-42a0-8657-5a99aa556133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75ece4b-f94kw"
Nov 03 18:05:53.285386 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:53.285331    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d33f7de6-f94f-4d92-8832-5bddc50358e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2610a6e1-9nwl5\" (UID: \"d33f7de6-f94f-4d92-8832-5bddc50358e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2610a6e1-9nwl5"
Nov 03 18:05:53.341031 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:53.340992    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-275g7\" (UniqueName: \"kubernetes.io/projected/d33f7de6-f94f-4d92-8832-5bddc50358e4-kube-api-access-275g7\") pod \"maas-api-revoke-sub-2610a6e1-9nwl5\" (UID: \"d33f7de6-f94f-4d92-8832-5bddc50358e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2610a6e1-9nwl5"
Nov 03 18:05:53.341186 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:53.341127    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl5m9\" (UniqueName: \"kubernetes.io/projected/510cff64-6154-42a0-8657-5a99aa556133-kube-api-access-wl5m9\") pod \"maas-api-revoke-sub-b75ece4b-f94kw\" (UID: \"510cff64-6154-42a0-8657-5a99aa556133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75ece4b-f94kw"
Nov 03 18:05:53.391010 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:53.390984    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75ece4b-f94kw"
Nov 03 18:05:53.441423 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:53.441388    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2610a6e1-9nwl5"
Nov 03 18:05:53.768277 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:53.768190    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b75ece4b-f94kw"]
Nov 03 18:05:53.772062 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:05:53.772037    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod510cff64_6154_42a0_8657_5a99aa556133.slice/crio-b0a356d488a527e8676fb205528b7042c8a60e9eaea5b8e28a17a855d54152c5 WatchSource:0}: Error finding container b0a356d488a527e8676fb205528b7042c8a60e9eaea5b8e28a17a855d54152c5: Status 404 returned error can't find the container with id b0a356d488a527e8676fb205528b7042c8a60e9eaea5b8e28a17a855d54152c5
Nov 03 18:05:53.845847 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:53.845805    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2610a6e1-9nwl5"]
Nov 03 18:05:53.850507 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:05:53.850473    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd33f7de6_f94f_4d92_8832_5bddc50358e4.slice/crio-f2caaa4af1d428139c883b73a69ea2f3f84f6775046d6bf0709fb8ae797342c8 WatchSource:0}: Error finding container f2caaa4af1d428139c883b73a69ea2f3f84f6775046d6bf0709fb8ae797342c8: Status 404 returned error can't find the container with id f2caaa4af1d428139c883b73a69ea2f3f84f6775046d6bf0709fb8ae797342c8
Nov 03 18:05:54.208133 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:54.208095    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2610a6e1-9nwl5" event={"ID":"d33f7de6-f94f-4d92-8832-5bddc50358e4","Type":"ContainerStarted","Data":"1df464538912ad4267a02a083c6ea97c796071a5ccfb40a5ab938499fa3b799c"}
Nov 03 18:05:54.208597 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:54.208149    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2610a6e1-9nwl5" event={"ID":"d33f7de6-f94f-4d92-8832-5bddc50358e4","Type":"ContainerStarted","Data":"f2caaa4af1d428139c883b73a69ea2f3f84f6775046d6bf0709fb8ae797342c8"}
Nov 03 18:05:54.209789 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:54.209758    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75ece4b-f94kw" event={"ID":"510cff64-6154-42a0-8657-5a99aa556133","Type":"ContainerStarted","Data":"3729e6fee38768f88fdd8d6bd886c8e4b1e061342613af9db14987a6425f3349"}
Nov 03 18:05:54.209918 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:54.209811    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75ece4b-f94kw" event={"ID":"510cff64-6154-42a0-8657-5a99aa556133","Type":"ContainerStarted","Data":"b0a356d488a527e8676fb205528b7042c8a60e9eaea5b8e28a17a855d54152c5"}
Nov 03 18:05:54.320059 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:54.319955    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2610a6e1-9nwl5" podStartSLOduration=1.319940773 podStartE2EDuration="1.319940773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:05:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:05:54.316113525 +0000 UTC m=+1193.332311593" watchObservedRunningTime="2026-11-03 18:05:54.319940773 +0000 UTC m=+1193.336138887"
Nov 03 18:05:54.371847 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:54.371820    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d7625f-rqgpz"
Nov 03 18:05:54.424001 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:54.423962    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e598e385-1a33-43c4-846b-8cc4e29f138e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e598e385-1a33-43c4-846b-8cc4e29f138e-maas-api-openshift-ca\") pod \"e598e385-1a33-43c4-846b-8cc4e29f138e\" (UID: \"e598e385-1a33-43c4-846b-8cc4e29f138e\") "
Nov 03 18:05:54.424218 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:54.424036    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e598e385-1a33-43c4-846b-8cc4e29f138e-kube-api-access-cnzgl\" (UniqueName: \"kubernetes.io/projected/e598e385-1a33-43c4-846b-8cc4e29f138e-kube-api-access-cnzgl\") pod \"e598e385-1a33-43c4-846b-8cc4e29f138e\" (UID: \"e598e385-1a33-43c4-846b-8cc4e29f138e\") "
Nov 03 18:05:54.424283 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:54.424254    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e598e385-1a33-43c4-846b-8cc4e29f138e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e598e385-1a33-43c4-846b-8cc4e29f138e-maas-api-xks-ca\") pod \"e598e385-1a33-43c4-846b-8cc4e29f138e\" (UID: \"e598e385-1a33-43c4-846b-8cc4e29f138e\") "
Nov 03 18:05:54.424753 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:54.424721    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e598e385-1a33-43c4-846b-8cc4e29f138e-maas-api-openshift-ca" pod "e598e385-1a33-43c4-846b-8cc4e29f138e" (UID: "e598e385-1a33-43c4-846b-8cc4e29f138e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:05:54.426629 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:54.426581    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e598e385-1a33-43c4-846b-8cc4e29f138e-maas-api-xks-ca" pod "e598e385-1a33-43c4-846b-8cc4e29f138e" (UID: "e598e385-1a33-43c4-846b-8cc4e29f138e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:05:54.426759 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:54.426634    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75ece4b-f94kw" podStartSLOduration=1.426604459 podStartE2EDuration="1.426604459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:05:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:05:54.423485228 +0000 UTC m=+1193.439683297" watchObservedRunningTime="2026-11-03 18:05:54.426604459 +0000 UTC m=+1193.442802531"
Nov 03 18:05:54.426759 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:54.426689    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e598e385-1a33-43c4-846b-8cc4e29f138e-kube-api-access-cnzgl" pod "e598e385-1a33-43c4-846b-8cc4e29f138e" (UID: "e598e385-1a33-43c4-846b-8cc4e29f138e"). InnerVolumeSpecName "kube-api-access-cnzgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:05:54.525948 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:54.525910    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e598e385-1a33-43c4-846b-8cc4e29f138e-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:05:54.525948 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:54.525947    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e598e385-1a33-43c4-846b-8cc4e29f138e-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:05:54.526150 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:54.525961    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnzgl\" (UniqueName: \"kubernetes.io/projected/e598e385-1a33-43c4-846b-8cc4e29f138e-kube-api-access-cnzgl\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:05:54.582672 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:05:54.582560    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:05:55.047868 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:55.047833    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b70309da-gtm7j"]
Nov 03 18:05:55.051645 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:55.051603    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b70309da-gtm7j"]
Nov 03 18:05:55.106455 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:55.106422    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b17a20a-d840-4bc2-a343-e84fea40beb5" path="/var/lib/kubelet/pods/5b17a20a-d840-4bc2-a343-e84fea40beb5/volumes"
Nov 03 18:05:55.192135 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:55.192089    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de9baa93-n7c5d"]
Nov 03 18:05:55.193391 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:55.193363    5315 cpu_manager.go:414] "containerMap: removing container" podUID="e598e385-1a33-43c4-846b-8cc4e29f138e" containerName="revoke-keys"
Nov 03 18:05:55.193391 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:55.193391    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e598e385-1a33-43c4-846b-8cc4e29f138e" containerName="revoke-keys"
Nov 03 18:05:55.193877 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:55.193849    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="e598e385-1a33-43c4-846b-8cc4e29f138e" containerName="revoke-keys"
Nov 03 18:05:55.197314 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:55.197298    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9baa93-n7c5d"
Nov 03 18:05:55.217705 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:55.217675    5315 generic.go:356] "Generic (PLEG): container finished" podID="d33f7de6-f94f-4d92-8832-5bddc50358e4" containerID="1df464538912ad4267a02a083c6ea97c796071a5ccfb40a5ab938499fa3b799c" exitCode=0
Nov 03 18:05:55.218032 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:55.217760    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2610a6e1-9nwl5" event={"ID":"d33f7de6-f94f-4d92-8832-5bddc50358e4","Type":"ContainerDied","Data":"1df464538912ad4267a02a083c6ea97c796071a5ccfb40a5ab938499fa3b799c"}
Nov 03 18:05:55.219391 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:55.219368    5315 generic.go:356] "Generic (PLEG): container finished" podID="510cff64-6154-42a0-8657-5a99aa556133" containerID="3729e6fee38768f88fdd8d6bd886c8e4b1e061342613af9db14987a6425f3349" exitCode=0
Nov 03 18:05:55.219494 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:55.219444    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75ece4b-f94kw" event={"ID":"510cff64-6154-42a0-8657-5a99aa556133","Type":"ContainerDied","Data":"3729e6fee38768f88fdd8d6bd886c8e4b1e061342613af9db14987a6425f3349"}
Nov 03 18:05:55.220952 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:55.220932    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d7625f-rqgpz"
Nov 03 18:05:55.221038 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:55.220935    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d7625f-rqgpz" event={"ID":"e598e385-1a33-43c4-846b-8cc4e29f138e","Type":"ContainerDied","Data":"505b2b3265a65f7976b8791566b14f1fff8e6062e2dbd7e54a2340ffb68777e7"}
Nov 03 18:05:55.221084 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:55.221038    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="505b2b3265a65f7976b8791566b14f1fff8e6062e2dbd7e54a2340ffb68777e7"
Nov 03 18:05:55.251571 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:55.251543    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de9baa93-n7c5d"]
Nov 03 18:05:55.335655 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:55.335545    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32d4f2d5-1cc6-438d-b734-00ba0ff2d187-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de9baa93-n7c5d\" (UID: \"32d4f2d5-1cc6-438d-b734-00ba0ff2d187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9baa93-n7c5d"
Nov 03 18:05:55.335829 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:55.335724    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shl5l\" (UniqueName: \"kubernetes.io/projected/32d4f2d5-1cc6-438d-b734-00ba0ff2d187-kube-api-access-shl5l\") pod \"maas-api-revoke-sub-de9baa93-n7c5d\" (UID: \"32d4f2d5-1cc6-438d-b734-00ba0ff2d187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9baa93-n7c5d"
Nov 03 18:05:55.335889 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:55.335823    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32d4f2d5-1cc6-438d-b734-00ba0ff2d187-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de9baa93-n7c5d\" (UID: \"32d4f2d5-1cc6-438d-b734-00ba0ff2d187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9baa93-n7c5d"
Nov 03 18:05:55.436576 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:55.436541    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shl5l\" (UniqueName: \"kubernetes.io/projected/32d4f2d5-1cc6-438d-b734-00ba0ff2d187-kube-api-access-shl5l\") pod \"maas-api-revoke-sub-de9baa93-n7c5d\" (UID: \"32d4f2d5-1cc6-438d-b734-00ba0ff2d187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9baa93-n7c5d"
Nov 03 18:05:55.436748 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:55.436631    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32d4f2d5-1cc6-438d-b734-00ba0ff2d187-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de9baa93-n7c5d\" (UID: \"32d4f2d5-1cc6-438d-b734-00ba0ff2d187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9baa93-n7c5d"
Nov 03 18:05:55.436784 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:55.436746    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32d4f2d5-1cc6-438d-b734-00ba0ff2d187-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de9baa93-n7c5d\" (UID: \"32d4f2d5-1cc6-438d-b734-00ba0ff2d187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9baa93-n7c5d"
Nov 03 18:05:55.437378 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:55.437344    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32d4f2d5-1cc6-438d-b734-00ba0ff2d187-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de9baa93-n7c5d\" (UID: \"32d4f2d5-1cc6-438d-b734-00ba0ff2d187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9baa93-n7c5d"
Nov 03 18:05:55.439023 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:55.438993    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32d4f2d5-1cc6-438d-b734-00ba0ff2d187-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de9baa93-n7c5d\" (UID: \"32d4f2d5-1cc6-438d-b734-00ba0ff2d187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9baa93-n7c5d"
Nov 03 18:05:55.494089 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:55.494058    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shl5l\" (UniqueName: \"kubernetes.io/projected/32d4f2d5-1cc6-438d-b734-00ba0ff2d187-kube-api-access-shl5l\") pod \"maas-api-revoke-sub-de9baa93-n7c5d\" (UID: \"32d4f2d5-1cc6-438d-b734-00ba0ff2d187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9baa93-n7c5d"
Nov 03 18:05:55.510557 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:55.510531    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9baa93-n7c5d"
Nov 03 18:05:55.886755 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:55.886721    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de9baa93-n7c5d"]
Nov 03 18:05:55.891399 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:05:55.891365    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32d4f2d5_1cc6_438d_b734_00ba0ff2d187.slice/crio-bb4bda34d11e3e7914bfc58527beb6596687698759d98152f3e9ee412db12e50 WatchSource:0}: Error finding container bb4bda34d11e3e7914bfc58527beb6596687698759d98152f3e9ee412db12e50: Status 404 returned error can't find the container with id bb4bda34d11e3e7914bfc58527beb6596687698759d98152f3e9ee412db12e50
Nov 03 18:05:56.047863 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:56.047829    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c73d193e-svwzg"]
Nov 03 18:05:56.051474 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:56.051436    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c73d193e-svwzg"]
Nov 03 18:05:56.229670 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:56.229639    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9baa93-n7c5d" event={"ID":"32d4f2d5-1cc6-438d-b734-00ba0ff2d187","Type":"ContainerStarted","Data":"bb4bda34d11e3e7914bfc58527beb6596687698759d98152f3e9ee412db12e50"}
Nov 03 18:05:56.428130 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:56.428101    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75ece4b-f94kw"
Nov 03 18:05:56.448787 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:56.448758    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2610a6e1-9nwl5"
Nov 03 18:05:56.554807 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:56.554767    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/510cff64-6154-42a0-8657-5a99aa556133-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/510cff64-6154-42a0-8657-5a99aa556133-maas-api-xks-ca\") pod \"510cff64-6154-42a0-8657-5a99aa556133\" (UID: \"510cff64-6154-42a0-8657-5a99aa556133\") "
Nov 03 18:05:56.554974 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:56.554858    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d33f7de6-f94f-4d92-8832-5bddc50358e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d33f7de6-f94f-4d92-8832-5bddc50358e4-maas-api-xks-ca\") pod \"d33f7de6-f94f-4d92-8832-5bddc50358e4\" (UID: \"d33f7de6-f94f-4d92-8832-5bddc50358e4\") "
Nov 03 18:05:56.554974 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:56.554902    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/510cff64-6154-42a0-8657-5a99aa556133-kube-api-access-wl5m9\" (UniqueName: \"kubernetes.io/projected/510cff64-6154-42a0-8657-5a99aa556133-kube-api-access-wl5m9\") pod \"510cff64-6154-42a0-8657-5a99aa556133\" (UID: \"510cff64-6154-42a0-8657-5a99aa556133\") "
Nov 03 18:05:56.554974 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:56.554940    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d33f7de6-f94f-4d92-8832-5bddc50358e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d33f7de6-f94f-4d92-8832-5bddc50358e4-maas-api-openshift-ca\") pod \"d33f7de6-f94f-4d92-8832-5bddc50358e4\" (UID: \"d33f7de6-f94f-4d92-8832-5bddc50358e4\") "
Nov 03 18:05:56.555179 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:56.555066    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/510cff64-6154-42a0-8657-5a99aa556133-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/510cff64-6154-42a0-8657-5a99aa556133-maas-api-openshift-ca\") pod \"510cff64-6154-42a0-8657-5a99aa556133\" (UID: \"510cff64-6154-42a0-8657-5a99aa556133\") "
Nov 03 18:05:56.555179 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:56.555159    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d33f7de6-f94f-4d92-8832-5bddc50358e4-kube-api-access-275g7\" (UniqueName: \"kubernetes.io/projected/d33f7de6-f94f-4d92-8832-5bddc50358e4-kube-api-access-275g7\") pod \"d33f7de6-f94f-4d92-8832-5bddc50358e4\" (UID: \"d33f7de6-f94f-4d92-8832-5bddc50358e4\") "
Nov 03 18:05:56.555515 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:56.555482    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d33f7de6-f94f-4d92-8832-5bddc50358e4-maas-api-openshift-ca" pod "d33f7de6-f94f-4d92-8832-5bddc50358e4" (UID: "d33f7de6-f94f-4d92-8832-5bddc50358e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:05:56.555882 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:56.555859    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d33f7de6-f94f-4d92-8832-5bddc50358e4-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:05:56.557654 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:56.557482    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/510cff64-6154-42a0-8657-5a99aa556133-maas-api-xks-ca" pod "510cff64-6154-42a0-8657-5a99aa556133" (UID: "510cff64-6154-42a0-8657-5a99aa556133"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:05:56.557654 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:56.557575    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d33f7de6-f94f-4d92-8832-5bddc50358e4-maas-api-xks-ca" pod "d33f7de6-f94f-4d92-8832-5bddc50358e4" (UID: "d33f7de6-f94f-4d92-8832-5bddc50358e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:05:56.557901 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:56.557877    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/510cff64-6154-42a0-8657-5a99aa556133-maas-api-openshift-ca" pod "510cff64-6154-42a0-8657-5a99aa556133" (UID: "510cff64-6154-42a0-8657-5a99aa556133"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:05:56.558254 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:56.558230    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d33f7de6-f94f-4d92-8832-5bddc50358e4-kube-api-access-275g7" pod "d33f7de6-f94f-4d92-8832-5bddc50358e4" (UID: "d33f7de6-f94f-4d92-8832-5bddc50358e4"). InnerVolumeSpecName "kube-api-access-275g7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:05:56.558837 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:56.558816    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/510cff64-6154-42a0-8657-5a99aa556133-kube-api-access-wl5m9" pod "510cff64-6154-42a0-8657-5a99aa556133" (UID: "510cff64-6154-42a0-8657-5a99aa556133"). InnerVolumeSpecName "kube-api-access-wl5m9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:05:56.657234 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:56.657193    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/510cff64-6154-42a0-8657-5a99aa556133-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:05:56.657472 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:56.657443    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d33f7de6-f94f-4d92-8832-5bddc50358e4-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:05:56.657593 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:56.657477    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl5m9\" (UniqueName: \"kubernetes.io/projected/510cff64-6154-42a0-8657-5a99aa556133-kube-api-access-wl5m9\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:05:56.657593 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:56.657495    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/510cff64-6154-42a0-8657-5a99aa556133-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:05:56.657593 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:56.657508    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-275g7\" (UniqueName: \"kubernetes.io/projected/d33f7de6-f94f-4d92-8832-5bddc50358e4-kube-api-access-275g7\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:05:57.104865 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:57.104831    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98ffe06e-3793-47ae-b331-b29dd02eaa4e" path="/var/lib/kubelet/pods/98ffe06e-3793-47ae-b331-b29dd02eaa4e/volumes"
Nov 03 18:05:57.239405 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:57.239316    5315 generic.go:356] "Generic (PLEG): container finished" podID="32d4f2d5-1cc6-438d-b734-00ba0ff2d187" containerID="e222ef1c0e6116539c82ce902e9ace8e8e2c830e3ebd1f4e0dba5354529203d0" exitCode=0
Nov 03 18:05:57.239819 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:57.239413    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9baa93-n7c5d" event={"ID":"32d4f2d5-1cc6-438d-b734-00ba0ff2d187","Type":"ContainerDied","Data":"e222ef1c0e6116539c82ce902e9ace8e8e2c830e3ebd1f4e0dba5354529203d0"}
Nov 03 18:05:57.240891 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:57.240865    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2610a6e1-9nwl5"
Nov 03 18:05:57.241012 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:57.240888    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2610a6e1-9nwl5" event={"ID":"d33f7de6-f94f-4d92-8832-5bddc50358e4","Type":"ContainerDied","Data":"f2caaa4af1d428139c883b73a69ea2f3f84f6775046d6bf0709fb8ae797342c8"}
Nov 03 18:05:57.241012 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:57.240930    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2caaa4af1d428139c883b73a69ea2f3f84f6775046d6bf0709fb8ae797342c8"
Nov 03 18:05:57.242386 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:57.242369    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75ece4b-f94kw"
Nov 03 18:05:57.242386 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:57.242375    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75ece4b-f94kw" event={"ID":"510cff64-6154-42a0-8657-5a99aa556133","Type":"ContainerDied","Data":"b0a356d488a527e8676fb205528b7042c8a60e9eaea5b8e28a17a855d54152c5"}
Nov 03 18:05:57.242588 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:57.242411    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0a356d488a527e8676fb205528b7042c8a60e9eaea5b8e28a17a855d54152c5"
Nov 03 18:05:58.246650 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:58.243399    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm"]
Nov 03 18:05:58.246650 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:58.244286    5315 cpu_manager.go:414] "containerMap: removing container" podUID="d33f7de6-f94f-4d92-8832-5bddc50358e4" containerName="revoke-keys"
Nov 03 18:05:58.246650 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:58.244307    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d33f7de6-f94f-4d92-8832-5bddc50358e4" containerName="revoke-keys"
Nov 03 18:05:58.246650 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:58.244339    5315 cpu_manager.go:414] "containerMap: removing container" podUID="510cff64-6154-42a0-8657-5a99aa556133" containerName="revoke-keys"
Nov 03 18:05:58.246650 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:58.244346    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="510cff64-6154-42a0-8657-5a99aa556133" containerName="revoke-keys"
Nov 03 18:05:58.246650 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:58.244467    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="510cff64-6154-42a0-8657-5a99aa556133" containerName="revoke-keys"
Nov 03 18:05:58.246650 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:58.244488    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="d33f7de6-f94f-4d92-8832-5bddc50358e4" containerName="revoke-keys"
Nov 03 18:05:58.250919 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:58.250895    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm"
Nov 03 18:05:58.301450 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:58.301416    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm"]
Nov 03 18:05:58.385151 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:58.385116    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/375f4e00-8c81-4de5-97bf-9be54ea028ed-istiod-ca-cert\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm\" (UID: \"375f4e00-8c81-4de5-97bf-9be54ea028ed\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm"
Nov 03 18:05:58.385330 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:58.385206    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/375f4e00-8c81-4de5-97bf-9be54ea028ed-istio-envoy\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm\" (UID: \"375f4e00-8c81-4de5-97bf-9be54ea028ed\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm"
Nov 03 18:05:58.385372 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:58.385339    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn444\" (UniqueName: \"kubernetes.io/projected/375f4e00-8c81-4de5-97bf-9be54ea028ed-kube-api-access-tn444\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm\" (UID: \"375f4e00-8c81-4de5-97bf-9be54ea028ed\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm"
Nov 03 18:05:58.385442 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:58.385423    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/375f4e00-8c81-4de5-97bf-9be54ea028ed-workload-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm\" (UID: \"375f4e00-8c81-4de5-97bf-9be54ea028ed\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm"
Nov 03 18:05:58.385523 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:58.385495    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/375f4e00-8c81-4de5-97bf-9be54ea028ed-istio-data\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm\" (UID: \"375f4e00-8c81-4de5-97bf-9be54ea028ed\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm"
Nov 03 18:05:58.385577 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:58.385548    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/375f4e00-8c81-4de5-97bf-9be54ea028ed-credential-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm\" (UID: \"375f4e00-8c81-4de5-97bf-9be54ea028ed\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm"
Nov 03 18:05:58.385668 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:58.385653    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/375f4e00-8c81-4de5-97bf-9be54ea028ed-istio-token\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm\" (UID: \"375f4e00-8c81-4de5-97bf-9be54ea028ed\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm"
Nov 03 18:05:58.385725 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:58.385698    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/375f4e00-8c81-4de5-97bf-9be54ea028ed-workload-certs\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm\" (UID: \"375f4e00-8c81-4de5-97bf-9be54ea028ed\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm"
Nov 03 18:05:58.385813 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:58.385799    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/375f4e00-8c81-4de5-97bf-9be54ea028ed-istio-podinfo\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm\" (UID: \"375f4e00-8c81-4de5-97bf-9be54ea028ed\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm"
Nov 03 18:05:58.435763 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:58.435738    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9baa93-n7c5d"
Nov 03 18:05:58.486741 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:58.486705    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/375f4e00-8c81-4de5-97bf-9be54ea028ed-istio-token\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm\" (UID: \"375f4e00-8c81-4de5-97bf-9be54ea028ed\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm"
Nov 03 18:05:58.486919 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:58.486760    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/375f4e00-8c81-4de5-97bf-9be54ea028ed-workload-certs\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm\" (UID: \"375f4e00-8c81-4de5-97bf-9be54ea028ed\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm"
Nov 03 18:05:58.486919 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:58.486821    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/375f4e00-8c81-4de5-97bf-9be54ea028ed-istio-podinfo\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm\" (UID: \"375f4e00-8c81-4de5-97bf-9be54ea028ed\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm"
Nov 03 18:05:58.487041 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:58.486940    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/375f4e00-8c81-4de5-97bf-9be54ea028ed-istiod-ca-cert\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm\" (UID: \"375f4e00-8c81-4de5-97bf-9be54ea028ed\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm"
Nov 03 18:05:58.487041 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:58.487022    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/375f4e00-8c81-4de5-97bf-9be54ea028ed-istio-envoy\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm\" (UID: \"375f4e00-8c81-4de5-97bf-9be54ea028ed\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm"
Nov 03 18:05:58.487141 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:58.487065    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn444\" (UniqueName: \"kubernetes.io/projected/375f4e00-8c81-4de5-97bf-9be54ea028ed-kube-api-access-tn444\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm\" (UID: \"375f4e00-8c81-4de5-97bf-9be54ea028ed\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm"
Nov 03 18:05:58.487141 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:58.487131    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/375f4e00-8c81-4de5-97bf-9be54ea028ed-workload-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm\" (UID: \"375f4e00-8c81-4de5-97bf-9be54ea028ed\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm"
Nov 03 18:05:58.487242 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:58.487166    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/375f4e00-8c81-4de5-97bf-9be54ea028ed-istio-data\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm\" (UID: \"375f4e00-8c81-4de5-97bf-9be54ea028ed\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm"
Nov 03 18:05:58.487242 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:58.487202    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/375f4e00-8c81-4de5-97bf-9be54ea028ed-credential-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm\" (UID: \"375f4e00-8c81-4de5-97bf-9be54ea028ed\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm"
Nov 03 18:05:58.487341 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:58.487246    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/375f4e00-8c81-4de5-97bf-9be54ea028ed-workload-certs\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm\" (UID: \"375f4e00-8c81-4de5-97bf-9be54ea028ed\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm"
Nov 03 18:05:58.487720 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:58.487689    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/375f4e00-8c81-4de5-97bf-9be54ea028ed-istio-data\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm\" (UID: \"375f4e00-8c81-4de5-97bf-9be54ea028ed\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm"
Nov 03 18:05:58.487828 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:58.487715    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/375f4e00-8c81-4de5-97bf-9be54ea028ed-istiod-ca-cert\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm\" (UID: \"375f4e00-8c81-4de5-97bf-9be54ea028ed\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm"
Nov 03 18:05:58.487828 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:58.487730    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/375f4e00-8c81-4de5-97bf-9be54ea028ed-credential-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm\" (UID: \"375f4e00-8c81-4de5-97bf-9be54ea028ed\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm"
Nov 03 18:05:58.487828 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:58.487719    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/375f4e00-8c81-4de5-97bf-9be54ea028ed-workload-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm\" (UID: \"375f4e00-8c81-4de5-97bf-9be54ea028ed\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm"
Nov 03 18:05:58.489326 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:58.489301    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/375f4e00-8c81-4de5-97bf-9be54ea028ed-istio-podinfo\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm\" (UID: \"375f4e00-8c81-4de5-97bf-9be54ea028ed\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm"
Nov 03 18:05:58.489436 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:58.489412    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/375f4e00-8c81-4de5-97bf-9be54ea028ed-istio-envoy\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm\" (UID: \"375f4e00-8c81-4de5-97bf-9be54ea028ed\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm"
Nov 03 18:05:58.544339 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:58.544310    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/375f4e00-8c81-4de5-97bf-9be54ea028ed-istio-token\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm\" (UID: \"375f4e00-8c81-4de5-97bf-9be54ea028ed\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm"
Nov 03 18:05:58.544596 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:58.544573    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn444\" (UniqueName: \"kubernetes.io/projected/375f4e00-8c81-4de5-97bf-9be54ea028ed-kube-api-access-tn444\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm\" (UID: \"375f4e00-8c81-4de5-97bf-9be54ea028ed\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm"
Nov 03 18:05:58.567000 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:58.566968    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm"
Nov 03 18:05:58.588371 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:58.588337    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32d4f2d5-1cc6-438d-b734-00ba0ff2d187-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32d4f2d5-1cc6-438d-b734-00ba0ff2d187-maas-api-openshift-ca\") pod \"32d4f2d5-1cc6-438d-b734-00ba0ff2d187\" (UID: \"32d4f2d5-1cc6-438d-b734-00ba0ff2d187\") "
Nov 03 18:05:58.588499 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:58.588472    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32d4f2d5-1cc6-438d-b734-00ba0ff2d187-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32d4f2d5-1cc6-438d-b734-00ba0ff2d187-maas-api-xks-ca\") pod \"32d4f2d5-1cc6-438d-b734-00ba0ff2d187\" (UID: \"32d4f2d5-1cc6-438d-b734-00ba0ff2d187\") "
Nov 03 18:05:58.588662 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:58.588639    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32d4f2d5-1cc6-438d-b734-00ba0ff2d187-kube-api-access-shl5l\" (UniqueName: \"kubernetes.io/projected/32d4f2d5-1cc6-438d-b734-00ba0ff2d187-kube-api-access-shl5l\") pod \"32d4f2d5-1cc6-438d-b734-00ba0ff2d187\" (UID: \"32d4f2d5-1cc6-438d-b734-00ba0ff2d187\") "
Nov 03 18:05:58.588763 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:58.588741    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32d4f2d5-1cc6-438d-b734-00ba0ff2d187-maas-api-openshift-ca" pod "32d4f2d5-1cc6-438d-b734-00ba0ff2d187" (UID: "32d4f2d5-1cc6-438d-b734-00ba0ff2d187"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:05:58.589183 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:58.589151    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32d4f2d5-1cc6-438d-b734-00ba0ff2d187-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:05:58.591239 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:58.591216    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32d4f2d5-1cc6-438d-b734-00ba0ff2d187-maas-api-xks-ca" pod "32d4f2d5-1cc6-438d-b734-00ba0ff2d187" (UID: "32d4f2d5-1cc6-438d-b734-00ba0ff2d187"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:05:58.591422 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:58.591399    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32d4f2d5-1cc6-438d-b734-00ba0ff2d187-kube-api-access-shl5l" pod "32d4f2d5-1cc6-438d-b734-00ba0ff2d187" (UID: "32d4f2d5-1cc6-438d-b734-00ba0ff2d187"). InnerVolumeSpecName "kube-api-access-shl5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:05:58.690289 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:58.690250    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32d4f2d5-1cc6-438d-b734-00ba0ff2d187-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:05:58.690289 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:58.690285    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shl5l\" (UniqueName: \"kubernetes.io/projected/32d4f2d5-1cc6-438d-b734-00ba0ff2d187-kube-api-access-shl5l\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:05:59.001438 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:59.001289    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm"]
Nov 03 18:05:59.005804 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:05:59.005779    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod375f4e00_8c81_4de5_97bf_9be54ea028ed.slice/crio-814bb639cb36888ce0cae90a0cb4115fb7a1ce49c1dc267297e540d19f293acf WatchSource:0}: Error finding container 814bb639cb36888ce0cae90a0cb4115fb7a1ce49c1dc267297e540d19f293acf: Status 404 returned error can't find the container with id 814bb639cb36888ce0cae90a0cb4115fb7a1ce49c1dc267297e540d19f293acf
Nov 03 18:05:59.008382 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:59.008341    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:05:59.008470 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:59.008409    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:05:59.008470 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:59.008437    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:05:59.265756 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:59.265667    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9baa93-n7c5d"
Nov 03 18:05:59.265756 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:59.265674    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9baa93-n7c5d" event={"ID":"32d4f2d5-1cc6-438d-b734-00ba0ff2d187","Type":"ContainerDied","Data":"bb4bda34d11e3e7914bfc58527beb6596687698759d98152f3e9ee412db12e50"}
Nov 03 18:05:59.265756 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:59.265728    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb4bda34d11e3e7914bfc58527beb6596687698759d98152f3e9ee412db12e50"
Nov 03 18:05:59.267694 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:59.267662    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm" event={"ID":"375f4e00-8c81-4de5-97bf-9be54ea028ed","Type":"ContainerStarted","Data":"79090806e6131c0c136e68a076d2e9ad0df849ff4976a5d41b06c0155b0be5f2"}
Nov 03 18:05:59.267837 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:59.267713    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm" event={"ID":"375f4e00-8c81-4de5-97bf-9be54ea028ed","Type":"ContainerStarted","Data":"814bb639cb36888ce0cae90a0cb4115fb7a1ce49c1dc267297e540d19f293acf"}
Nov 03 18:05:59.384979 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:59.384924    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm" podStartSLOduration=1.384907268 podStartE2EDuration="1.384907268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:05:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:05:59.380084703 +0000 UTC m=+1198.396282783" watchObservedRunningTime="2026-11-03 18:05:59.384907268 +0000 UTC m=+1198.401105424"
Nov 03 18:05:59.571797 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:05:59.570993    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm"
Nov 03 18:06:00.568706 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:00.568670    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 03 18:06:00.569079 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:00.568736    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm" podUID="375f4e00-8c81-4de5-97bf-9be54ea028ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 03 18:06:01.286556 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:01.286525    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-97p65_c29d200a-c917-46e5-9cf5-21f4d952e470/ovn-acl-logging/0.log"
Nov 03 18:06:01.289431 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:01.289400    5315 scope.go:122] "RemoveContainer" containerID="c7f990e9ea27cdf2f40787b6896ec7ed42dee1b36ddfa7f836333050ca3075f9"
Nov 03 18:06:01.301000 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:01.300968    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-97p65_c29d200a-c917-46e5-9cf5-21f4d952e470/ovn-acl-logging/0.log"
Nov 03 18:06:01.301420 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:01.301396    5315 scope.go:122] "RemoveContainer" containerID="fad7732a6044a3f44384900b623768236ddbf2c33b3d16715c39bc3155307870"
Nov 03 18:06:01.311764 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:01.311739    5315 scope.go:122] "RemoveContainer" containerID="4d61ad13581a65fa19a21783fc9d56b898e66afe353e6cdb35d4f95ee3ffec83"
Nov 03 18:06:01.567776 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:01.567688    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 03 18:06:01.567776 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:01.567744    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm" podUID="375f4e00-8c81-4de5-97bf-9be54ea028ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 03 18:06:02.054105 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:02.054063    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d421e35-kq2q2"]
Nov 03 18:06:02.058102 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:02.058066    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d421e35-kq2q2"]
Nov 03 18:06:02.567988 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:02.567945    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 03 18:06:02.568178 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:02.568027    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm" podUID="375f4e00-8c81-4de5-97bf-9be54ea028ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 03 18:06:02.910947 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:02.910854    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc80bea1-xm6fj"]
Nov 03 18:06:02.911733 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:02.911708    5315 cpu_manager.go:414] "containerMap: removing container" podUID="32d4f2d5-1cc6-438d-b734-00ba0ff2d187" containerName="revoke-keys"
Nov 03 18:06:02.911733 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:02.911732    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32d4f2d5-1cc6-438d-b734-00ba0ff2d187" containerName="revoke-keys"
Nov 03 18:06:02.911913 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:02.911901    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="32d4f2d5-1cc6-438d-b734-00ba0ff2d187" containerName="revoke-keys"
Nov 03 18:06:02.917420 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:02.917395    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc80bea1-xm6fj"
Nov 03 18:06:02.966591 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:02.966556    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:06:02.966833 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:02.966814    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:06:03.012829 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:03.012792    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:06:03.013001 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:03.012868    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:06:03.019052 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:03.019016    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc80bea1-xm6fj"]
Nov 03 18:06:03.072220 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:03.072185    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf5wv\" (UniqueName: \"kubernetes.io/projected/c9a18cf4-7b26-47b3-8295-eb63eaa924d1-kube-api-access-zf5wv\") pod \"maas-api-revoke-sub-bc80bea1-xm6fj\" (UID: \"c9a18cf4-7b26-47b3-8295-eb63eaa924d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc80bea1-xm6fj"
Nov 03 18:06:03.072590 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:03.072303    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9a18cf4-7b26-47b3-8295-eb63eaa924d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc80bea1-xm6fj\" (UID: \"c9a18cf4-7b26-47b3-8295-eb63eaa924d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc80bea1-xm6fj"
Nov 03 18:06:03.072590 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:03.072366    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9a18cf4-7b26-47b3-8295-eb63eaa924d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc80bea1-xm6fj\" (UID: \"c9a18cf4-7b26-47b3-8295-eb63eaa924d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc80bea1-xm6fj"
Nov 03 18:06:03.105713 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:03.105680    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea648b10-b2ec-49ec-8b3d-cc10668990b1" path="/var/lib/kubelet/pods/ea648b10-b2ec-49ec-8b3d-cc10668990b1/volumes"
Nov 03 18:06:03.173341 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:03.173248    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9a18cf4-7b26-47b3-8295-eb63eaa924d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc80bea1-xm6fj\" (UID: \"c9a18cf4-7b26-47b3-8295-eb63eaa924d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc80bea1-xm6fj"
Nov 03 18:06:03.173508 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:03.173352    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9a18cf4-7b26-47b3-8295-eb63eaa924d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc80bea1-xm6fj\" (UID: \"c9a18cf4-7b26-47b3-8295-eb63eaa924d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc80bea1-xm6fj"
Nov 03 18:06:03.173508 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:03.173411    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf5wv\" (UniqueName: \"kubernetes.io/projected/c9a18cf4-7b26-47b3-8295-eb63eaa924d1-kube-api-access-zf5wv\") pod \"maas-api-revoke-sub-bc80bea1-xm6fj\" (UID: \"c9a18cf4-7b26-47b3-8295-eb63eaa924d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc80bea1-xm6fj"
Nov 03 18:06:03.174173 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:03.174141    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9a18cf4-7b26-47b3-8295-eb63eaa924d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc80bea1-xm6fj\" (UID: \"c9a18cf4-7b26-47b3-8295-eb63eaa924d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc80bea1-xm6fj"
Nov 03 18:06:03.175935 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:03.175906    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9a18cf4-7b26-47b3-8295-eb63eaa924d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc80bea1-xm6fj\" (UID: \"c9a18cf4-7b26-47b3-8295-eb63eaa924d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc80bea1-xm6fj"
Nov 03 18:06:03.256541 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:03.256507    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf5wv\" (UniqueName: \"kubernetes.io/projected/c9a18cf4-7b26-47b3-8295-eb63eaa924d1-kube-api-access-zf5wv\") pod \"maas-api-revoke-sub-bc80bea1-xm6fj\" (UID: \"c9a18cf4-7b26-47b3-8295-eb63eaa924d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc80bea1-xm6fj"
Nov 03 18:06:03.262166 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:03.262134    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc80bea1-xm6fj"
Nov 03 18:06:03.567644 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:03.567327    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 03 18:06:03.567644 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:03.567377    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm" podUID="375f4e00-8c81-4de5-97bf-9be54ea028ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 03 18:06:03.671040 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:03.670962    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc80bea1-xm6fj"]
Nov 03 18:06:03.676648 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:06:03.676589    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9a18cf4_7b26_47b3_8295_eb63eaa924d1.slice/crio-78b1db87a29377adffe03459ae274708a8a74ae16e0014734dde7d8837dcaf13 WatchSource:0}: Error finding container 78b1db87a29377adffe03459ae274708a8a74ae16e0014734dde7d8837dcaf13: Status 404 returned error can't find the container with id 78b1db87a29377adffe03459ae274708a8a74ae16e0014734dde7d8837dcaf13
Nov 03 18:06:04.314435 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:04.314402    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc80bea1-xm6fj" event={"ID":"c9a18cf4-7b26-47b3-8295-eb63eaa924d1","Type":"ContainerStarted","Data":"0371632aa5e4fefeb7113787793526666eef8ed3cc33936c5a39c5d52c4754b5"}
Nov 03 18:06:04.314870 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:04.314458    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc80bea1-xm6fj" event={"ID":"c9a18cf4-7b26-47b3-8295-eb63eaa924d1","Type":"ContainerStarted","Data":"78b1db87a29377adffe03459ae274708a8a74ae16e0014734dde7d8837dcaf13"}
Nov 03 18:06:04.439635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:04.423602    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc80bea1-xm6fj" podStartSLOduration=2.423588621 podStartE2EDuration="2.423588621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:06:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:06:04.422506814 +0000 UTC m=+1203.438704884" watchObservedRunningTime="2026-11-03 18:06:04.423588621 +0000 UTC m=+1203.439786735"
Nov 03 18:06:04.568175 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:04.568055    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 03 18:06:04.568175 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:04.568133    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm" podUID="375f4e00-8c81-4de5-97bf-9be54ea028ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 03 18:06:05.323072 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:05.323035    5315 generic.go:356] "Generic (PLEG): container finished" podID="c9a18cf4-7b26-47b3-8295-eb63eaa924d1" containerID="0371632aa5e4fefeb7113787793526666eef8ed3cc33936c5a39c5d52c4754b5" exitCode=0
Nov 03 18:06:05.323474 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:05.323120    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc80bea1-xm6fj" event={"ID":"c9a18cf4-7b26-47b3-8295-eb63eaa924d1","Type":"ContainerDied","Data":"0371632aa5e4fefeb7113787793526666eef8ed3cc33936c5a39c5d52c4754b5"}
Nov 03 18:06:05.568184 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:05.568151    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 03 18:06:05.568436 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:05.568203    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm" podUID="375f4e00-8c81-4de5-97bf-9be54ea028ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 03 18:06:06.491056 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:06.491028    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc80bea1-xm6fj"
Nov 03 18:06:06.513238 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:06.513199    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9a18cf4-7b26-47b3-8295-eb63eaa924d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9a18cf4-7b26-47b3-8295-eb63eaa924d1-maas-api-openshift-ca\") pod \"c9a18cf4-7b26-47b3-8295-eb63eaa924d1\" (UID: \"c9a18cf4-7b26-47b3-8295-eb63eaa924d1\") "
Nov 03 18:06:06.513386 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:06.513289    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9a18cf4-7b26-47b3-8295-eb63eaa924d1-kube-api-access-zf5wv\" (UniqueName: \"kubernetes.io/projected/c9a18cf4-7b26-47b3-8295-eb63eaa924d1-kube-api-access-zf5wv\") pod \"c9a18cf4-7b26-47b3-8295-eb63eaa924d1\" (UID: \"c9a18cf4-7b26-47b3-8295-eb63eaa924d1\") "
Nov 03 18:06:06.513386 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:06.513354    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9a18cf4-7b26-47b3-8295-eb63eaa924d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9a18cf4-7b26-47b3-8295-eb63eaa924d1-maas-api-xks-ca\") pod \"c9a18cf4-7b26-47b3-8295-eb63eaa924d1\" (UID: \"c9a18cf4-7b26-47b3-8295-eb63eaa924d1\") "
Nov 03 18:06:06.513529 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:06.513506    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9a18cf4-7b26-47b3-8295-eb63eaa924d1-maas-api-openshift-ca" pod "c9a18cf4-7b26-47b3-8295-eb63eaa924d1" (UID: "c9a18cf4-7b26-47b3-8295-eb63eaa924d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:06:06.513870 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:06.513847    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9a18cf4-7b26-47b3-8295-eb63eaa924d1-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:06.515467 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:06.515442    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9a18cf4-7b26-47b3-8295-eb63eaa924d1-kube-api-access-zf5wv" pod "c9a18cf4-7b26-47b3-8295-eb63eaa924d1" (UID: "c9a18cf4-7b26-47b3-8295-eb63eaa924d1"). InnerVolumeSpecName "kube-api-access-zf5wv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:06:06.515549 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:06.515512    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9a18cf4-7b26-47b3-8295-eb63eaa924d1-maas-api-xks-ca" pod "c9a18cf4-7b26-47b3-8295-eb63eaa924d1" (UID: "c9a18cf4-7b26-47b3-8295-eb63eaa924d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:06:06.571958 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:06.571926    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm"
Nov 03 18:06:06.572359 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:06.572329    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm"
Nov 03 18:06:06.572975 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:06.572956    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm"
Nov 03 18:06:06.615500 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:06.615419    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf5wv\" (UniqueName: \"kubernetes.io/projected/c9a18cf4-7b26-47b3-8295-eb63eaa924d1-kube-api-access-zf5wv\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:06.615500 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:06.615446    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9a18cf4-7b26-47b3-8295-eb63eaa924d1-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:07.338884 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:07.338853    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc80bea1-xm6fj"
Nov 03 18:06:07.339084 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:07.338880    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc80bea1-xm6fj" event={"ID":"c9a18cf4-7b26-47b3-8295-eb63eaa924d1","Type":"ContainerDied","Data":"78b1db87a29377adffe03459ae274708a8a74ae16e0014734dde7d8837dcaf13"}
Nov 03 18:06:07.339084 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:07.338944    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78b1db87a29377adffe03459ae274708a8a74ae16e0014734dde7d8837dcaf13"
Nov 03 18:06:08.306155 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:08.306112    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07f3d541-btzqt"]
Nov 03 18:06:08.307021 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:08.306997    5315 cpu_manager.go:414] "containerMap: removing container" podUID="c9a18cf4-7b26-47b3-8295-eb63eaa924d1" containerName="revoke-keys"
Nov 03 18:06:08.307124 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:08.307025    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9a18cf4-7b26-47b3-8295-eb63eaa924d1" containerName="revoke-keys"
Nov 03 18:06:08.307220 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:08.307204    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9a18cf4-7b26-47b3-8295-eb63eaa924d1" containerName="revoke-keys"
Nov 03 18:06:08.319259 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:08.319230    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f3d541-btzqt"
Nov 03 18:06:08.342850 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:08.342815    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn7k4\" (UniqueName: \"kubernetes.io/projected/ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d-kube-api-access-jn7k4\") pod \"maas-api-revoke-sub-07f3d541-btzqt\" (UID: \"ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f3d541-btzqt"
Nov 03 18:06:08.343182 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:08.343158    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07f3d541-btzqt\" (UID: \"ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f3d541-btzqt"
Nov 03 18:06:08.343482 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:08.343463    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07f3d541-btzqt\" (UID: \"ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f3d541-btzqt"
Nov 03 18:06:08.395088 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:08.395048    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07f3d541-btzqt"]
Nov 03 18:06:08.404057 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:08.404027    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:06:08.404560 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:08.404387    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:06:08.408813 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:08.404459    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:06:08.422863 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:08.422832    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:06:08.446523 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:08.446480    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn7k4\" (UniqueName: \"kubernetes.io/projected/ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d-kube-api-access-jn7k4\") pod \"maas-api-revoke-sub-07f3d541-btzqt\" (UID: \"ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f3d541-btzqt"
Nov 03 18:06:08.493233 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:08.489868    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07f3d541-btzqt\" (UID: \"ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f3d541-btzqt"
Nov 03 18:06:08.507635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:08.493751    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07f3d541-btzqt\" (UID: \"ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f3d541-btzqt"
Nov 03 18:06:08.509632 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:08.501853    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07f3d541-btzqt\" (UID: \"ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f3d541-btzqt"
Nov 03 18:06:08.548483 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:08.548446    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07f3d541-btzqt\" (UID: \"ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f3d541-btzqt"
Nov 03 18:06:08.571266 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:08.571162    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn7k4\" (UniqueName: \"kubernetes.io/projected/ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d-kube-api-access-jn7k4\") pod \"maas-api-revoke-sub-07f3d541-btzqt\" (UID: \"ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f3d541-btzqt"
Nov 03 18:06:08.706867 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:08.706825    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f3d541-btzqt"
Nov 03 18:06:09.658651 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:09.657885    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07f3d541-btzqt"]
Nov 03 18:06:09.694390 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:06:09.694352    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac972aa2_f7a1_435f_8fd7_4814ab5e0f0d.slice/crio-3541a3416dfe9cbb7ed4b5a35227ffefd52a724e8e3220cd5f941d047d76ac0a WatchSource:0}: Error finding container 3541a3416dfe9cbb7ed4b5a35227ffefd52a724e8e3220cd5f941d047d76ac0a: Status 404 returned error can't find the container with id 3541a3416dfe9cbb7ed4b5a35227ffefd52a724e8e3220cd5f941d047d76ac0a
Nov 03 18:06:09.983773 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:09.983726    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7" podUID="69760dba-faf9-4c08-895c-ad70fe95fd49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:10.036085 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:10.036036    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c" podUID="8ac59539-8ac1-466e-8d1b-b2be4c42c708" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:10.073932 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:10.073901    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88e3834e-kwk9d"]
Nov 03 18:06:10.095635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:10.089643    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88e3834e-kwk9d"]
Nov 03 18:06:10.636240 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:10.636195    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f3d541-btzqt" event={"ID":"ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d","Type":"ContainerStarted","Data":"68668f9bf2d9483d82f7396878b4345af80cf2bffc1a140ef85e1a69103abfc8"}
Nov 03 18:06:10.636523 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:10.636494    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f3d541-btzqt" event={"ID":"ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d","Type":"ContainerStarted","Data":"3541a3416dfe9cbb7ed4b5a35227ffefd52a724e8e3220cd5f941d047d76ac0a"}
Nov 03 18:06:10.756728 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:10.756649    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f3d541-btzqt" podStartSLOduration=2.756628334 podStartE2EDuration="2.756628334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:06:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:06:10.754838269 +0000 UTC m=+1209.771036342" watchObservedRunningTime="2026-11-03 18:06:10.756628334 +0000 UTC m=+1209.772826402"
Nov 03 18:06:11.053931 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:11.053857    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c53a27e-stzjc"]
Nov 03 18:06:11.058781 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:11.058749    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a599aa06-5kd6z"]
Nov 03 18:06:11.094486 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:11.094450    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c53a27e-stzjc"]
Nov 03 18:06:11.108373 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:11.108329    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fd12a08-7286-4611-a553-c59ce5e54326" path="/var/lib/kubelet/pods/1fd12a08-7286-4611-a553-c59ce5e54326/volumes"
Nov 03 18:06:11.108911 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:11.108885    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58247d8d-0abc-4f70-8da3-ff79907ccd82" path="/var/lib/kubelet/pods/58247d8d-0abc-4f70-8da3-ff79907ccd82/volumes"
Nov 03 18:06:11.109659 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:11.109531    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a599aa06-5kd6z"]
Nov 03 18:06:11.149087 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:06:11.149047    5315 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-a599aa06-5kd6z\" is forbidden: User \"system:node:ip-10-0-128-204.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-128-204.us-west-2.compute.internal' and this object" podUID="091bbf48-d823-4b71-bc36-eb10bc68d8f7" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a599aa06-5kd6z"
Nov 03 18:06:11.644899 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:11.644854    5315 generic.go:356] "Generic (PLEG): container finished" podID="ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d" containerID="68668f9bf2d9483d82f7396878b4345af80cf2bffc1a140ef85e1a69103abfc8" exitCode=0
Nov 03 18:06:11.644899 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:11.644888    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f3d541-btzqt" event={"ID":"ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d","Type":"ContainerDied","Data":"68668f9bf2d9483d82f7396878b4345af80cf2bffc1a140ef85e1a69103abfc8"}
Nov 03 18:06:11.938725 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:11.935396    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6814726-rtxxz"]
Nov 03 18:06:11.941045 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:11.941012    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6814726-rtxxz"
Nov 03 18:06:11.967929 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:11.967899    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6814726-rtxxz"]
Nov 03 18:06:12.027346 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.025085    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq2gd\" (UniqueName: \"kubernetes.io/projected/3224261c-8cfa-4f55-90e6-cbbd099f8de7-kube-api-access-qq2gd\") pod \"maas-api-revoke-sub-b6814726-rtxxz\" (UID: \"3224261c-8cfa-4f55-90e6-cbbd099f8de7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6814726-rtxxz"
Nov 03 18:06:12.027346 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.025246    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3224261c-8cfa-4f55-90e6-cbbd099f8de7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6814726-rtxxz\" (UID: \"3224261c-8cfa-4f55-90e6-cbbd099f8de7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6814726-rtxxz"
Nov 03 18:06:12.027346 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.025349    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3224261c-8cfa-4f55-90e6-cbbd099f8de7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6814726-rtxxz\" (UID: \"3224261c-8cfa-4f55-90e6-cbbd099f8de7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6814726-rtxxz"
Nov 03 18:06:12.126561 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.126527    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq2gd\" (UniqueName: \"kubernetes.io/projected/3224261c-8cfa-4f55-90e6-cbbd099f8de7-kube-api-access-qq2gd\") pod \"maas-api-revoke-sub-b6814726-rtxxz\" (UID: \"3224261c-8cfa-4f55-90e6-cbbd099f8de7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6814726-rtxxz"
Nov 03 18:06:12.126770 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.126747    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3224261c-8cfa-4f55-90e6-cbbd099f8de7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6814726-rtxxz\" (UID: \"3224261c-8cfa-4f55-90e6-cbbd099f8de7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6814726-rtxxz"
Nov 03 18:06:12.126895 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.126870    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3224261c-8cfa-4f55-90e6-cbbd099f8de7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6814726-rtxxz\" (UID: \"3224261c-8cfa-4f55-90e6-cbbd099f8de7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6814726-rtxxz"
Nov 03 18:06:12.127360 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.127335    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3224261c-8cfa-4f55-90e6-cbbd099f8de7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6814726-rtxxz\" (UID: \"3224261c-8cfa-4f55-90e6-cbbd099f8de7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6814726-rtxxz"
Nov 03 18:06:12.129156 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.129132    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3224261c-8cfa-4f55-90e6-cbbd099f8de7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6814726-rtxxz\" (UID: \"3224261c-8cfa-4f55-90e6-cbbd099f8de7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6814726-rtxxz"
Nov 03 18:06:12.183577 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.183544    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq2gd\" (UniqueName: \"kubernetes.io/projected/3224261c-8cfa-4f55-90e6-cbbd099f8de7-kube-api-access-qq2gd\") pod \"maas-api-revoke-sub-b6814726-rtxxz\" (UID: \"3224261c-8cfa-4f55-90e6-cbbd099f8de7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6814726-rtxxz"
Nov 03 18:06:12.230637 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.230531    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs"]
Nov 03 18:06:12.234912 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.234892    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs"
Nov 03 18:06:12.272837 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.272751    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6814726-rtxxz"
Nov 03 18:06:12.335654 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.332201    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1977c777-abc0-4d10-a0ee-3985dd2be5be-istiod-ca-cert\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs\" (UID: \"1977c777-abc0-4d10-a0ee-3985dd2be5be\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs"
Nov 03 18:06:12.335654 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.332279    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1977c777-abc0-4d10-a0ee-3985dd2be5be-istio-podinfo\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs\" (UID: \"1977c777-abc0-4d10-a0ee-3985dd2be5be\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs"
Nov 03 18:06:12.335654 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.332318    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzcrd\" (UniqueName: \"kubernetes.io/projected/1977c777-abc0-4d10-a0ee-3985dd2be5be-kube-api-access-qzcrd\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs\" (UID: \"1977c777-abc0-4d10-a0ee-3985dd2be5be\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs"
Nov 03 18:06:12.335654 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.332416    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1977c777-abc0-4d10-a0ee-3985dd2be5be-workload-certs\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs\" (UID: \"1977c777-abc0-4d10-a0ee-3985dd2be5be\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs"
Nov 03 18:06:12.335654 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.332489    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1977c777-abc0-4d10-a0ee-3985dd2be5be-credential-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs\" (UID: \"1977c777-abc0-4d10-a0ee-3985dd2be5be\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs"
Nov 03 18:06:12.335654 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.332625    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1977c777-abc0-4d10-a0ee-3985dd2be5be-istio-token\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs\" (UID: \"1977c777-abc0-4d10-a0ee-3985dd2be5be\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs"
Nov 03 18:06:12.335654 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.332664    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1977c777-abc0-4d10-a0ee-3985dd2be5be-istio-data\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs\" (UID: \"1977c777-abc0-4d10-a0ee-3985dd2be5be\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs"
Nov 03 18:06:12.335654 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.332737    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1977c777-abc0-4d10-a0ee-3985dd2be5be-workload-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs\" (UID: \"1977c777-abc0-4d10-a0ee-3985dd2be5be\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs"
Nov 03 18:06:12.335654 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.332758    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1977c777-abc0-4d10-a0ee-3985dd2be5be-istio-envoy\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs\" (UID: \"1977c777-abc0-4d10-a0ee-3985dd2be5be\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs"
Nov 03 18:06:12.382154 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.381059    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs"]
Nov 03 18:06:12.434035 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.433993    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1977c777-abc0-4d10-a0ee-3985dd2be5be-istio-token\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs\" (UID: \"1977c777-abc0-4d10-a0ee-3985dd2be5be\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs"
Nov 03 18:06:12.434208 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.434048    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1977c777-abc0-4d10-a0ee-3985dd2be5be-istio-data\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs\" (UID: \"1977c777-abc0-4d10-a0ee-3985dd2be5be\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs"
Nov 03 18:06:12.434208 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.434107    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1977c777-abc0-4d10-a0ee-3985dd2be5be-workload-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs\" (UID: \"1977c777-abc0-4d10-a0ee-3985dd2be5be\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs"
Nov 03 18:06:12.434208 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.434129    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1977c777-abc0-4d10-a0ee-3985dd2be5be-istio-envoy\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs\" (UID: \"1977c777-abc0-4d10-a0ee-3985dd2be5be\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs"
Nov 03 18:06:12.434208 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.434173    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1977c777-abc0-4d10-a0ee-3985dd2be5be-istiod-ca-cert\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs\" (UID: \"1977c777-abc0-4d10-a0ee-3985dd2be5be\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs"
Nov 03 18:06:12.434423 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.434216    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1977c777-abc0-4d10-a0ee-3985dd2be5be-istio-podinfo\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs\" (UID: \"1977c777-abc0-4d10-a0ee-3985dd2be5be\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs"
Nov 03 18:06:12.434423 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.434274    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzcrd\" (UniqueName: \"kubernetes.io/projected/1977c777-abc0-4d10-a0ee-3985dd2be5be-kube-api-access-qzcrd\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs\" (UID: \"1977c777-abc0-4d10-a0ee-3985dd2be5be\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs"
Nov 03 18:06:12.434423 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.434370    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1977c777-abc0-4d10-a0ee-3985dd2be5be-workload-certs\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs\" (UID: \"1977c777-abc0-4d10-a0ee-3985dd2be5be\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs"
Nov 03 18:06:12.434566 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.434424    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1977c777-abc0-4d10-a0ee-3985dd2be5be-istio-data\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs\" (UID: \"1977c777-abc0-4d10-a0ee-3985dd2be5be\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs"
Nov 03 18:06:12.434566 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.434439    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1977c777-abc0-4d10-a0ee-3985dd2be5be-credential-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs\" (UID: \"1977c777-abc0-4d10-a0ee-3985dd2be5be\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs"
Nov 03 18:06:12.434907 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.434881    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1977c777-abc0-4d10-a0ee-3985dd2be5be-credential-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs\" (UID: \"1977c777-abc0-4d10-a0ee-3985dd2be5be\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs"
Nov 03 18:06:12.435246 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.435220    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1977c777-abc0-4d10-a0ee-3985dd2be5be-workload-certs\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs\" (UID: \"1977c777-abc0-4d10-a0ee-3985dd2be5be\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs"
Nov 03 18:06:12.435387 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.435326    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1977c777-abc0-4d10-a0ee-3985dd2be5be-istiod-ca-cert\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs\" (UID: \"1977c777-abc0-4d10-a0ee-3985dd2be5be\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs"
Nov 03 18:06:12.435484 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.435464    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1977c777-abc0-4d10-a0ee-3985dd2be5be-workload-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs\" (UID: \"1977c777-abc0-4d10-a0ee-3985dd2be5be\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs"
Nov 03 18:06:12.436904 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.436885    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1977c777-abc0-4d10-a0ee-3985dd2be5be-istio-envoy\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs\" (UID: \"1977c777-abc0-4d10-a0ee-3985dd2be5be\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs"
Nov 03 18:06:12.437743 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.437725    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1977c777-abc0-4d10-a0ee-3985dd2be5be-istio-podinfo\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs\" (UID: \"1977c777-abc0-4d10-a0ee-3985dd2be5be\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs"
Nov 03 18:06:12.492313 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.492248    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1977c777-abc0-4d10-a0ee-3985dd2be5be-istio-token\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs\" (UID: \"1977c777-abc0-4d10-a0ee-3985dd2be5be\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs"
Nov 03 18:06:12.492438 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.492329    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzcrd\" (UniqueName: \"kubernetes.io/projected/1977c777-abc0-4d10-a0ee-3985dd2be5be-kube-api-access-qzcrd\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs\" (UID: \"1977c777-abc0-4d10-a0ee-3985dd2be5be\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs"
Nov 03 18:06:12.548742 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.548700    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m"]
Nov 03 18:06:12.548927 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.548831    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs"
Nov 03 18:06:12.553146 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.553105    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m"
Nov 03 18:06:12.622509 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.622475    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m"]
Nov 03 18:06:12.645116 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.638233    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df589ab9-3107-4b55-9195-6ee315bd3965-credential-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m\" (UID: \"df589ab9-3107-4b55-9195-6ee315bd3965\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m"
Nov 03 18:06:12.645116 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.639091    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkr29\" (UniqueName: \"kubernetes.io/projected/df589ab9-3107-4b55-9195-6ee315bd3965-kube-api-access-gkr29\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m\" (UID: \"df589ab9-3107-4b55-9195-6ee315bd3965\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m"
Nov 03 18:06:12.645116 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.639138    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df589ab9-3107-4b55-9195-6ee315bd3965-istio-data\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m\" (UID: \"df589ab9-3107-4b55-9195-6ee315bd3965\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m"
Nov 03 18:06:12.645116 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.639172    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df589ab9-3107-4b55-9195-6ee315bd3965-workload-certs\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m\" (UID: \"df589ab9-3107-4b55-9195-6ee315bd3965\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m"
Nov 03 18:06:12.645116 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.639240    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df589ab9-3107-4b55-9195-6ee315bd3965-istio-token\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m\" (UID: \"df589ab9-3107-4b55-9195-6ee315bd3965\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m"
Nov 03 18:06:12.645116 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.639322    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df589ab9-3107-4b55-9195-6ee315bd3965-istio-envoy\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m\" (UID: \"df589ab9-3107-4b55-9195-6ee315bd3965\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m"
Nov 03 18:06:12.645116 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.639350    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df589ab9-3107-4b55-9195-6ee315bd3965-istiod-ca-cert\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m\" (UID: \"df589ab9-3107-4b55-9195-6ee315bd3965\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m"
Nov 03 18:06:12.645116 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.639387    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df589ab9-3107-4b55-9195-6ee315bd3965-workload-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m\" (UID: \"df589ab9-3107-4b55-9195-6ee315bd3965\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m"
Nov 03 18:06:12.645116 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.639442    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df589ab9-3107-4b55-9195-6ee315bd3965-istio-podinfo\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m\" (UID: \"df589ab9-3107-4b55-9195-6ee315bd3965\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m"
Nov 03 18:06:12.701814 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.699541    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6814726-rtxxz"]
Nov 03 18:06:12.744112 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.744046    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df589ab9-3107-4b55-9195-6ee315bd3965-istiod-ca-cert\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m\" (UID: \"df589ab9-3107-4b55-9195-6ee315bd3965\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m"
Nov 03 18:06:12.744112 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.744097    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df589ab9-3107-4b55-9195-6ee315bd3965-workload-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m\" (UID: \"df589ab9-3107-4b55-9195-6ee315bd3965\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m"
Nov 03 18:06:12.744273 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.744138    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df589ab9-3107-4b55-9195-6ee315bd3965-istio-podinfo\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m\" (UID: \"df589ab9-3107-4b55-9195-6ee315bd3965\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m"
Nov 03 18:06:12.744307 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.744270    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df589ab9-3107-4b55-9195-6ee315bd3965-credential-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m\" (UID: \"df589ab9-3107-4b55-9195-6ee315bd3965\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m"
Nov 03 18:06:12.744344 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.744305    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkr29\" (UniqueName: \"kubernetes.io/projected/df589ab9-3107-4b55-9195-6ee315bd3965-kube-api-access-gkr29\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m\" (UID: \"df589ab9-3107-4b55-9195-6ee315bd3965\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m"
Nov 03 18:06:12.744344 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.744327    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df589ab9-3107-4b55-9195-6ee315bd3965-istio-data\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m\" (UID: \"df589ab9-3107-4b55-9195-6ee315bd3965\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m"
Nov 03 18:06:12.744404 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.744347    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df589ab9-3107-4b55-9195-6ee315bd3965-workload-certs\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m\" (UID: \"df589ab9-3107-4b55-9195-6ee315bd3965\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m"
Nov 03 18:06:12.744404 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.744395    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df589ab9-3107-4b55-9195-6ee315bd3965-istio-token\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m\" (UID: \"df589ab9-3107-4b55-9195-6ee315bd3965\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m"
Nov 03 18:06:12.744465 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.744447    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df589ab9-3107-4b55-9195-6ee315bd3965-istio-envoy\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m\" (UID: \"df589ab9-3107-4b55-9195-6ee315bd3965\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m"
Nov 03 18:06:12.745043 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.745016    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df589ab9-3107-4b55-9195-6ee315bd3965-credential-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m\" (UID: \"df589ab9-3107-4b55-9195-6ee315bd3965\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m"
Nov 03 18:06:12.745487 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.745465    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df589ab9-3107-4b55-9195-6ee315bd3965-istiod-ca-cert\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m\" (UID: \"df589ab9-3107-4b55-9195-6ee315bd3965\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m"
Nov 03 18:06:12.745767 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.745749    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df589ab9-3107-4b55-9195-6ee315bd3965-workload-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m\" (UID: \"df589ab9-3107-4b55-9195-6ee315bd3965\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m"
Nov 03 18:06:12.745906 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.745858    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df589ab9-3107-4b55-9195-6ee315bd3965-workload-certs\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m\" (UID: \"df589ab9-3107-4b55-9195-6ee315bd3965\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m"
Nov 03 18:06:12.745906 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.745872    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df589ab9-3107-4b55-9195-6ee315bd3965-istio-data\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m\" (UID: \"df589ab9-3107-4b55-9195-6ee315bd3965\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m"
Nov 03 18:06:12.750469 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.750446    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df589ab9-3107-4b55-9195-6ee315bd3965-istio-envoy\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m\" (UID: \"df589ab9-3107-4b55-9195-6ee315bd3965\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m"
Nov 03 18:06:12.750994 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.750969    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df589ab9-3107-4b55-9195-6ee315bd3965-istio-podinfo\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m\" (UID: \"df589ab9-3107-4b55-9195-6ee315bd3965\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m"
Nov 03 18:06:12.803558 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.803504    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df589ab9-3107-4b55-9195-6ee315bd3965-istio-token\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m\" (UID: \"df589ab9-3107-4b55-9195-6ee315bd3965\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m"
Nov 03 18:06:12.803775 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.803757    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkr29\" (UniqueName: \"kubernetes.io/projected/df589ab9-3107-4b55-9195-6ee315bd3965-kube-api-access-gkr29\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m\" (UID: \"df589ab9-3107-4b55-9195-6ee315bd3965\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m"
Nov 03 18:06:12.817057 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.817033    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f3d541-btzqt"
Nov 03 18:06:12.845836 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.845805    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d-maas-api-openshift-ca\") pod \"ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d\" (UID: \"ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d\") "
Nov 03 18:06:12.846029 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.845854    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d-kube-api-access-jn7k4\" (UniqueName: \"kubernetes.io/projected/ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d-kube-api-access-jn7k4\") pod \"ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d\" (UID: \"ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d\") "
Nov 03 18:06:12.846029 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.845877    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d-maas-api-xks-ca\") pod \"ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d\" (UID: \"ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d\") "
Nov 03 18:06:12.846227 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.846203    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d-maas-api-openshift-ca" pod "ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d" (UID: "ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:06:12.846483 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.846462    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:12.847926 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.847904    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d-maas-api-xks-ca" pod "ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d" (UID: "ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:06:12.848355 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.848331    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d-kube-api-access-jn7k4" pod "ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d" (UID: "ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d"). InnerVolumeSpecName "kube-api-access-jn7k4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:06:12.870846 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.870817    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m"
Nov 03 18:06:12.947229 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.947199    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn7k4\" (UniqueName: \"kubernetes.io/projected/ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d-kube-api-access-jn7k4\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:12.947229 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.947225    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:12.948654 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.948563    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:06:12.948758 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.948691    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:06:12.948758 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.948739    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:06:12.949919 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:12.949893    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs"]
Nov 03 18:06:13.106425 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.106388    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="091bbf48-d823-4b71-bc36-eb10bc68d8f7" path="/var/lib/kubelet/pods/091bbf48-d823-4b71-bc36-eb10bc68d8f7/volumes"
Nov 03 18:06:13.260256 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.260159    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m"]
Nov 03 18:06:13.265950 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:06:13.265919    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf589ab9_3107_4b55_9195_6ee315bd3965.slice/crio-ed094bb4ba311957cfff8b401e4da4a06623b700392c9e4c87fda3c07b164dc7 WatchSource:0}: Error finding container ed094bb4ba311957cfff8b401e4da4a06623b700392c9e4c87fda3c07b164dc7: Status 404 returned error can't find the container with id ed094bb4ba311957cfff8b401e4da4a06623b700392c9e4c87fda3c07b164dc7
Nov 03 18:06:13.268598 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.268561    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:06:13.268708 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.268665    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:06:13.268708 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.268702    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:06:13.565879 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.565846    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daea96db-846wr"]
Nov 03 18:06:13.566433 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.566416    5315 cpu_manager.go:414] "containerMap: removing container" podUID="ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d" containerName="revoke-keys"
Nov 03 18:06:13.566433 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.566431    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d" containerName="revoke-keys"
Nov 03 18:06:13.566687 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.566673    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d" containerName="revoke-keys"
Nov 03 18:06:13.570941 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.570915    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daea96db-846wr"
Nov 03 18:06:13.629962 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.629930    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daea96db-846wr"]
Nov 03 18:06:13.656672 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.656633    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r6w4\" (UniqueName: \"kubernetes.io/projected/327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98-kube-api-access-7r6w4\") pod \"maas-api-revoke-sub-daea96db-846wr\" (UID: \"327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daea96db-846wr"
Nov 03 18:06:13.657046 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.657023    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daea96db-846wr\" (UID: \"327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daea96db-846wr"
Nov 03 18:06:13.657159 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.657031    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-227429c2-kl7ck"]
Nov 03 18:06:13.659741 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.657238    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daea96db-846wr\" (UID: \"327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daea96db-846wr"
Nov 03 18:06:13.662036 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.662010    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227429c2-kl7ck"
Nov 03 18:06:13.669340 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.669307    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f3d541-btzqt" event={"ID":"ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d","Type":"ContainerDied","Data":"3541a3416dfe9cbb7ed4b5a35227ffefd52a724e8e3220cd5f941d047d76ac0a"}
Nov 03 18:06:13.669470 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.669362    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3541a3416dfe9cbb7ed4b5a35227ffefd52a724e8e3220cd5f941d047d76ac0a"
Nov 03 18:06:13.669470 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.669424    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07f3d541-btzqt"
Nov 03 18:06:13.672583 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.672554    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6814726-rtxxz" event={"ID":"3224261c-8cfa-4f55-90e6-cbbd099f8de7","Type":"ContainerStarted","Data":"67915755c38e19d2b6c61f43435a204693c61a24d80eb679495b98d3ef8649cb"}
Nov 03 18:06:13.672860 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.672813    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6814726-rtxxz" event={"ID":"3224261c-8cfa-4f55-90e6-cbbd099f8de7","Type":"ContainerStarted","Data":"2990983aa101106f7209be1e2d6bcfc5a0edb61aca20edaf9f815f3f6dd0e6b2"}
Nov 03 18:06:13.675284 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.674958    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m" event={"ID":"df589ab9-3107-4b55-9195-6ee315bd3965","Type":"ContainerStarted","Data":"1b8d55022722af293a4b45330a88ed9187b8c288ed428fc844fb82dca87afc04"}
Nov 03 18:06:13.675284 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.675007    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m" event={"ID":"df589ab9-3107-4b55-9195-6ee315bd3965","Type":"ContainerStarted","Data":"ed094bb4ba311957cfff8b401e4da4a06623b700392c9e4c87fda3c07b164dc7"}
Nov 03 18:06:13.678495 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.678466    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs" event={"ID":"1977c777-abc0-4d10-a0ee-3985dd2be5be","Type":"ContainerStarted","Data":"20c9b60fc479b95e3bf4ceb30bf0668e9fd0cb63001a00378e9709e988fe2cd1"}
Nov 03 18:06:13.678604 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.678519    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs" event={"ID":"1977c777-abc0-4d10-a0ee-3985dd2be5be","Type":"ContainerStarted","Data":"c4a19d1b5b171de39fb44cf77c2eb9d57aea0d52f0ab8cbf8d241983f203c241"}
Nov 03 18:06:13.722135 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.722090    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-227429c2-kl7ck"]
Nov 03 18:06:13.760809 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.760766    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daea96db-846wr\" (UID: \"327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daea96db-846wr"
Nov 03 18:06:13.760984 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.760885    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0e395dc-0519-4588-aa3e-aa9067700361-maas-api-xks-ca\") pod \"maas-api-revoke-sub-227429c2-kl7ck\" (UID: \"f0e395dc-0519-4588-aa3e-aa9067700361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227429c2-kl7ck"
Nov 03 18:06:13.761118 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.761091    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r6w4\" (UniqueName: \"kubernetes.io/projected/327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98-kube-api-access-7r6w4\") pod \"maas-api-revoke-sub-daea96db-846wr\" (UID: \"327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daea96db-846wr"
Nov 03 18:06:13.761240 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.761137    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daea96db-846wr\" (UID: \"327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daea96db-846wr"
Nov 03 18:06:13.761240 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.761163    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wfzf\" (UniqueName: \"kubernetes.io/projected/f0e395dc-0519-4588-aa3e-aa9067700361-kube-api-access-9wfzf\") pod \"maas-api-revoke-sub-227429c2-kl7ck\" (UID: \"f0e395dc-0519-4588-aa3e-aa9067700361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227429c2-kl7ck"
Nov 03 18:06:13.762328 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.761239    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0e395dc-0519-4588-aa3e-aa9067700361-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-227429c2-kl7ck\" (UID: \"f0e395dc-0519-4588-aa3e-aa9067700361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227429c2-kl7ck"
Nov 03 18:06:13.764462 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.764432    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daea96db-846wr\" (UID: \"327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daea96db-846wr"
Nov 03 18:06:13.765449 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.765430    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daea96db-846wr\" (UID: \"327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daea96db-846wr"
Nov 03 18:06:13.791927 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.791897    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj"]
Nov 03 18:06:13.795868 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.795780    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m" podStartSLOduration=1.795765786 podStartE2EDuration="1.795765786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:06:13.790641316 +0000 UTC m=+1212.806839382" watchObservedRunningTime="2026-11-03 18:06:13.795765786 +0000 UTC m=+1212.811963854"
Nov 03 18:06:13.797691 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.797670    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj"
Nov 03 18:06:13.812024 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.811995    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp"]
Nov 03 18:06:13.816084 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.816020    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp"
Nov 03 18:06:13.822770 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.822745    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r6w4\" (UniqueName: \"kubernetes.io/projected/327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98-kube-api-access-7r6w4\") pod \"maas-api-revoke-sub-daea96db-846wr\" (UID: \"327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daea96db-846wr"
Nov 03 18:06:13.864234 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.864197    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj"]
Nov 03 18:06:13.865515 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.865477    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59f3868f-6b95-4d02-afe7-4e5228cf37c3-istio-token\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp\" (UID: \"59f3868f-6b95-4d02-afe7-4e5228cf37c3\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp"
Nov 03 18:06:13.865840 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.865817    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd9b0713-a9c7-4344-8b32-487d235f9a72-istio-token\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj\" (UID: \"bd9b0713-a9c7-4344-8b32-487d235f9a72\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj"
Nov 03 18:06:13.866050 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.866032    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59f3868f-6b95-4d02-afe7-4e5228cf37c3-istio-envoy\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp\" (UID: \"59f3868f-6b95-4d02-afe7-4e5228cf37c3\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp"
Nov 03 18:06:13.866255 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.866229    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wfzf\" (UniqueName: \"kubernetes.io/projected/f0e395dc-0519-4588-aa3e-aa9067700361-kube-api-access-9wfzf\") pod \"maas-api-revoke-sub-227429c2-kl7ck\" (UID: \"f0e395dc-0519-4588-aa3e-aa9067700361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227429c2-kl7ck"
Nov 03 18:06:13.866377 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.866285    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd9b0713-a9c7-4344-8b32-487d235f9a72-workload-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj\" (UID: \"bd9b0713-a9c7-4344-8b32-487d235f9a72\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj"
Nov 03 18:06:13.866377 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.866326    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd9b0713-a9c7-4344-8b32-487d235f9a72-workload-certs\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj\" (UID: \"bd9b0713-a9c7-4344-8b32-487d235f9a72\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj"
Nov 03 18:06:13.866377 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.866354    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0e395dc-0519-4588-aa3e-aa9067700361-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-227429c2-kl7ck\" (UID: \"f0e395dc-0519-4588-aa3e-aa9067700361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227429c2-kl7ck"
Nov 03 18:06:13.866544 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.866387    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59f3868f-6b95-4d02-afe7-4e5228cf37c3-istio-podinfo\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp\" (UID: \"59f3868f-6b95-4d02-afe7-4e5228cf37c3\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp"
Nov 03 18:06:13.866544 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.866432    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cnqk\" (UniqueName: \"kubernetes.io/projected/bd9b0713-a9c7-4344-8b32-487d235f9a72-kube-api-access-9cnqk\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj\" (UID: \"bd9b0713-a9c7-4344-8b32-487d235f9a72\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj"
Nov 03 18:06:13.866544 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.866478    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd9b0713-a9c7-4344-8b32-487d235f9a72-istio-envoy\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj\" (UID: \"bd9b0713-a9c7-4344-8b32-487d235f9a72\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj"
Nov 03 18:06:13.866544 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.866503    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd9b0713-a9c7-4344-8b32-487d235f9a72-istiod-ca-cert\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj\" (UID: \"bd9b0713-a9c7-4344-8b32-487d235f9a72\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj"
Nov 03 18:06:13.866544 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.866528    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59f3868f-6b95-4d02-afe7-4e5228cf37c3-istio-data\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp\" (UID: \"59f3868f-6b95-4d02-afe7-4e5228cf37c3\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp"
Nov 03 18:06:13.866844 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.866562    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f3868f-6b95-4d02-afe7-4e5228cf37c3-workload-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp\" (UID: \"59f3868f-6b95-4d02-afe7-4e5228cf37c3\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp"
Nov 03 18:06:13.866844 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.866588    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59f3868f-6b95-4d02-afe7-4e5228cf37c3-istiod-ca-cert\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp\" (UID: \"59f3868f-6b95-4d02-afe7-4e5228cf37c3\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp"
Nov 03 18:06:13.866844 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.866639    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd9b0713-a9c7-4344-8b32-487d235f9a72-credential-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj\" (UID: \"bd9b0713-a9c7-4344-8b32-487d235f9a72\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj"
Nov 03 18:06:13.866844 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.866665    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd9b0713-a9c7-4344-8b32-487d235f9a72-istio-data\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj\" (UID: \"bd9b0713-a9c7-4344-8b32-487d235f9a72\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj"
Nov 03 18:06:13.867705 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.867681    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f3868f-6b95-4d02-afe7-4e5228cf37c3-credential-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp\" (UID: \"59f3868f-6b95-4d02-afe7-4e5228cf37c3\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp"
Nov 03 18:06:13.867836 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.867724    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59f3868f-6b95-4d02-afe7-4e5228cf37c3-workload-certs\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp\" (UID: \"59f3868f-6b95-4d02-afe7-4e5228cf37c3\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp"
Nov 03 18:06:13.867836 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.867778    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpbd9\" (UniqueName: \"kubernetes.io/projected/59f3868f-6b95-4d02-afe7-4e5228cf37c3-kube-api-access-lpbd9\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp\" (UID: \"59f3868f-6b95-4d02-afe7-4e5228cf37c3\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp"
Nov 03 18:06:13.867836 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.867831    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd9b0713-a9c7-4344-8b32-487d235f9a72-istio-podinfo\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj\" (UID: \"bd9b0713-a9c7-4344-8b32-487d235f9a72\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj"
Nov 03 18:06:13.868022 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.867893    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0e395dc-0519-4588-aa3e-aa9067700361-maas-api-xks-ca\") pod \"maas-api-revoke-sub-227429c2-kl7ck\" (UID: \"f0e395dc-0519-4588-aa3e-aa9067700361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227429c2-kl7ck"
Nov 03 18:06:13.869135 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.869109    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0e395dc-0519-4588-aa3e-aa9067700361-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-227429c2-kl7ck\" (UID: \"f0e395dc-0519-4588-aa3e-aa9067700361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227429c2-kl7ck"
Nov 03 18:06:13.872104 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.872026    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m"
Nov 03 18:06:13.873451 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.873427    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0e395dc-0519-4588-aa3e-aa9067700361-maas-api-xks-ca\") pod \"maas-api-revoke-sub-227429c2-kl7ck\" (UID: \"f0e395dc-0519-4588-aa3e-aa9067700361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227429c2-kl7ck"
Nov 03 18:06:13.882539 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.882505    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp"]
Nov 03 18:06:13.890213 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.889968    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daea96db-846wr"
Nov 03 18:06:13.906548 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.906318    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6814726-rtxxz" podStartSLOduration=2.906304509 podStartE2EDuration="2.906304509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:06:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:06:13.900842189 +0000 UTC m=+1212.917040258" watchObservedRunningTime="2026-11-03 18:06:13.906304509 +0000 UTC m=+1212.922502577"
Nov 03 18:06:13.925920 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.925884    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wfzf\" (UniqueName: \"kubernetes.io/projected/f0e395dc-0519-4588-aa3e-aa9067700361-kube-api-access-9wfzf\") pod \"maas-api-revoke-sub-227429c2-kl7ck\" (UID: \"f0e395dc-0519-4588-aa3e-aa9067700361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227429c2-kl7ck"
Nov 03 18:06:13.969811 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.969771    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd9b0713-a9c7-4344-8b32-487d235f9a72-istio-token\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj\" (UID: \"bd9b0713-a9c7-4344-8b32-487d235f9a72\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj"
Nov 03 18:06:13.970584 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.970563    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59f3868f-6b95-4d02-afe7-4e5228cf37c3-istio-envoy\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp\" (UID: \"59f3868f-6b95-4d02-afe7-4e5228cf37c3\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp"
Nov 03 18:06:13.971178 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.971160    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd9b0713-a9c7-4344-8b32-487d235f9a72-workload-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj\" (UID: \"bd9b0713-a9c7-4344-8b32-487d235f9a72\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj"
Nov 03 18:06:13.971748 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.971731    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd9b0713-a9c7-4344-8b32-487d235f9a72-workload-certs\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj\" (UID: \"bd9b0713-a9c7-4344-8b32-487d235f9a72\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj"
Nov 03 18:06:13.972160 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.972146    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59f3868f-6b95-4d02-afe7-4e5228cf37c3-istio-podinfo\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp\" (UID: \"59f3868f-6b95-4d02-afe7-4e5228cf37c3\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp"
Nov 03 18:06:13.972675 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.972656    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cnqk\" (UniqueName: \"kubernetes.io/projected/bd9b0713-a9c7-4344-8b32-487d235f9a72-kube-api-access-9cnqk\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj\" (UID: \"bd9b0713-a9c7-4344-8b32-487d235f9a72\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj"
Nov 03 18:06:13.973084 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.973067    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd9b0713-a9c7-4344-8b32-487d235f9a72-istio-envoy\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj\" (UID: \"bd9b0713-a9c7-4344-8b32-487d235f9a72\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj"
Nov 03 18:06:13.973207 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.973193    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd9b0713-a9c7-4344-8b32-487d235f9a72-istiod-ca-cert\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj\" (UID: \"bd9b0713-a9c7-4344-8b32-487d235f9a72\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj"
Nov 03 18:06:13.973301 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.973288    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59f3868f-6b95-4d02-afe7-4e5228cf37c3-istio-data\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp\" (UID: \"59f3868f-6b95-4d02-afe7-4e5228cf37c3\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp"
Nov 03 18:06:13.973402 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.973386    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f3868f-6b95-4d02-afe7-4e5228cf37c3-workload-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp\" (UID: \"59f3868f-6b95-4d02-afe7-4e5228cf37c3\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp"
Nov 03 18:06:13.973491 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.973479    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59f3868f-6b95-4d02-afe7-4e5228cf37c3-istiod-ca-cert\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp\" (UID: \"59f3868f-6b95-4d02-afe7-4e5228cf37c3\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp"
Nov 03 18:06:13.973587 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.973571    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd9b0713-a9c7-4344-8b32-487d235f9a72-credential-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj\" (UID: \"bd9b0713-a9c7-4344-8b32-487d235f9a72\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj"
Nov 03 18:06:13.973689 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.973674    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd9b0713-a9c7-4344-8b32-487d235f9a72-istio-data\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj\" (UID: \"bd9b0713-a9c7-4344-8b32-487d235f9a72\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj"
Nov 03 18:06:13.973883 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.973869    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f3868f-6b95-4d02-afe7-4e5228cf37c3-credential-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp\" (UID: \"59f3868f-6b95-4d02-afe7-4e5228cf37c3\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp"
Nov 03 18:06:13.973974 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.973962    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59f3868f-6b95-4d02-afe7-4e5228cf37c3-workload-certs\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp\" (UID: \"59f3868f-6b95-4d02-afe7-4e5228cf37c3\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp"
Nov 03 18:06:13.974080 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.974066    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpbd9\" (UniqueName: \"kubernetes.io/projected/59f3868f-6b95-4d02-afe7-4e5228cf37c3-kube-api-access-lpbd9\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp\" (UID: \"59f3868f-6b95-4d02-afe7-4e5228cf37c3\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp"
Nov 03 18:06:13.974190 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.974177    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd9b0713-a9c7-4344-8b32-487d235f9a72-istio-podinfo\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj\" (UID: \"bd9b0713-a9c7-4344-8b32-487d235f9a72\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj"
Nov 03 18:06:13.974372 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.974359    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59f3868f-6b95-4d02-afe7-4e5228cf37c3-istio-token\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp\" (UID: \"59f3868f-6b95-4d02-afe7-4e5228cf37c3\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp"
Nov 03 18:06:13.974742 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.972097    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd9b0713-a9c7-4344-8b32-487d235f9a72-workload-certs\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj\" (UID: \"bd9b0713-a9c7-4344-8b32-487d235f9a72\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj"
Nov 03 18:06:13.978241 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.975322    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd9b0713-a9c7-4344-8b32-487d235f9a72-credential-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj\" (UID: \"bd9b0713-a9c7-4344-8b32-487d235f9a72\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj"
Nov 03 18:06:13.978241 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.971661    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd9b0713-a9c7-4344-8b32-487d235f9a72-workload-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj\" (UID: \"bd9b0713-a9c7-4344-8b32-487d235f9a72\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj"
Nov 03 18:06:13.978241 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.975929    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd9b0713-a9c7-4344-8b32-487d235f9a72-istiod-ca-cert\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj\" (UID: \"bd9b0713-a9c7-4344-8b32-487d235f9a72\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj"
Nov 03 18:06:13.978241 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.976239    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59f3868f-6b95-4d02-afe7-4e5228cf37c3-istio-data\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp\" (UID: \"59f3868f-6b95-4d02-afe7-4e5228cf37c3\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp"
Nov 03 18:06:13.978241 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.976447    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f3868f-6b95-4d02-afe7-4e5228cf37c3-workload-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp\" (UID: \"59f3868f-6b95-4d02-afe7-4e5228cf37c3\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp"
Nov 03 18:06:13.978241 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.977076    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59f3868f-6b95-4d02-afe7-4e5228cf37c3-istiod-ca-cert\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp\" (UID: \"59f3868f-6b95-4d02-afe7-4e5228cf37c3\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp"
Nov 03 18:06:13.978241 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.977296    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59f3868f-6b95-4d02-afe7-4e5228cf37c3-workload-certs\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp\" (UID: \"59f3868f-6b95-4d02-afe7-4e5228cf37c3\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp"
Nov 03 18:06:13.978241 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.977496    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd9b0713-a9c7-4344-8b32-487d235f9a72-istio-data\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj\" (UID: \"bd9b0713-a9c7-4344-8b32-487d235f9a72\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj"
Nov 03 18:06:13.978241 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.977727    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f3868f-6b95-4d02-afe7-4e5228cf37c3-credential-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp\" (UID: \"59f3868f-6b95-4d02-afe7-4e5228cf37c3\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp"
Nov 03 18:06:13.979970 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.979898    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59f3868f-6b95-4d02-afe7-4e5228cf37c3-istio-envoy\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp\" (UID: \"59f3868f-6b95-4d02-afe7-4e5228cf37c3\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp"
Nov 03 18:06:13.995388 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.994171    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd9b0713-a9c7-4344-8b32-487d235f9a72-istio-podinfo\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj\" (UID: \"bd9b0713-a9c7-4344-8b32-487d235f9a72\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj"
Nov 03 18:06:13.995388 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.995321    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd9b0713-a9c7-4344-8b32-487d235f9a72-istio-envoy\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj\" (UID: \"bd9b0713-a9c7-4344-8b32-487d235f9a72\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj"
Nov 03 18:06:13.999897 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:13.999677    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227429c2-kl7ck"
Nov 03 18:06:14.003040 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:14.002970    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59f3868f-6b95-4d02-afe7-4e5228cf37c3-istio-podinfo\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp\" (UID: \"59f3868f-6b95-4d02-afe7-4e5228cf37c3\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp"
Nov 03 18:06:14.052188 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:14.048729    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd9b0713-a9c7-4344-8b32-487d235f9a72-istio-token\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj\" (UID: \"bd9b0713-a9c7-4344-8b32-487d235f9a72\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj"
Nov 03 18:06:14.059936 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:14.059870    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1048ceff-6bsxw"]
Nov 03 18:06:14.066237 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:14.066155    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1048ceff-6bsxw"]
Nov 03 18:06:14.069709 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:14.069602    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cnqk\" (UniqueName: \"kubernetes.io/projected/bd9b0713-a9c7-4344-8b32-487d235f9a72-kube-api-access-9cnqk\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj\" (UID: \"bd9b0713-a9c7-4344-8b32-487d235f9a72\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj"
Nov 03 18:06:14.070468 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:14.070416    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59f3868f-6b95-4d02-afe7-4e5228cf37c3-istio-token\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp\" (UID: \"59f3868f-6b95-4d02-afe7-4e5228cf37c3\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp"
Nov 03 18:06:14.071129 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:14.071093    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpbd9\" (UniqueName: \"kubernetes.io/projected/59f3868f-6b95-4d02-afe7-4e5228cf37c3-kube-api-access-lpbd9\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp\" (UID: \"59f3868f-6b95-4d02-afe7-4e5228cf37c3\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp"
Nov 03 18:06:14.119235 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:14.118976    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj"
Nov 03 18:06:14.137171 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:14.136947    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp"
Nov 03 18:06:14.304645 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:14.302435    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daea96db-846wr"]
Nov 03 18:06:14.310534 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:06:14.310453    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod327ab1b6_a43b_4f94_ba1b_15f4cc3b4e98.slice/crio-9ee6c8291538ce0eb1c54a8d71f2c4bb8a002e3da0e1360d4e08ca09e15de2f5 WatchSource:0}: Error finding container 9ee6c8291538ce0eb1c54a8d71f2c4bb8a002e3da0e1360d4e08ca09e15de2f5: Status 404 returned error can't find the container with id 9ee6c8291538ce0eb1c54a8d71f2c4bb8a002e3da0e1360d4e08ca09e15de2f5
Nov 03 18:06:14.487482 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:14.487402    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-227429c2-kl7ck"]
Nov 03 18:06:14.487782 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:14.487707    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs" podStartSLOduration=2.487695488 podStartE2EDuration="2.487695488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:06:14.455149056 +0000 UTC m=+1213.471347139" watchObservedRunningTime="2026-11-03 18:06:14.487695488 +0000 UTC m=+1213.503893557"
Nov 03 18:06:14.494538 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:06:14.494504    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0e395dc_0519_4588_aa3e_aa9067700361.slice/crio-b849d7487959cb5330f25373d93b9d894b7071c31f5f391360b0872c896b1e94 WatchSource:0}: Error finding container b849d7487959cb5330f25373d93b9d894b7071c31f5f391360b0872c896b1e94: Status 404 returned error can't find the container with id b849d7487959cb5330f25373d93b9d894b7071c31f5f391360b0872c896b1e94
Nov 03 18:06:14.564055 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:14.564023    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs"
Nov 03 18:06:14.576712 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:14.575671    5315 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 18:06:14.576712 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:14.575737    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs" podUID="1977c777-abc0-4d10-a0ee-3985dd2be5be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 18:06:14.678632 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:06:14.672247    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd9b0713_a9c7_4344_8b32_487d235f9a72.slice/crio-419b63653f48325ff899a3d1f5a17153275403b2696ce6ebc1e57cec7f63f109 WatchSource:0}: Error finding container 419b63653f48325ff899a3d1f5a17153275403b2696ce6ebc1e57cec7f63f109: Status 404 returned error can't find the container with id 419b63653f48325ff899a3d1f5a17153275403b2696ce6ebc1e57cec7f63f109
Nov 03 18:06:14.678632 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:14.674966    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:06:14.678632 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:14.675053    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:06:14.678632 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:14.675095    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:06:14.680852 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:14.680484    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj"]
Nov 03 18:06:14.686762 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:14.686717    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp"]
Nov 03 18:06:14.691314 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:06:14.691236    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59f3868f_6b95_4d02_afe7_4e5228cf37c3.slice/crio-c7424cb8369383e8bc1200d6fe00b6f49a5c3bd44a0acead0723bd2511374fa0 WatchSource:0}: Error finding container c7424cb8369383e8bc1200d6fe00b6f49a5c3bd44a0acead0723bd2511374fa0: Status 404 returned error can't find the container with id c7424cb8369383e8bc1200d6fe00b6f49a5c3bd44a0acead0723bd2511374fa0
Nov 03 18:06:14.692928 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:14.692905    5315 generic.go:356] "Generic (PLEG): container finished" podID="3224261c-8cfa-4f55-90e6-cbbd099f8de7" containerID="67915755c38e19d2b6c61f43435a204693c61a24d80eb679495b98d3ef8649cb" exitCode=0
Nov 03 18:06:14.693023 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:14.692986    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6814726-rtxxz" event={"ID":"3224261c-8cfa-4f55-90e6-cbbd099f8de7","Type":"ContainerDied","Data":"67915755c38e19d2b6c61f43435a204693c61a24d80eb679495b98d3ef8649cb"}
Nov 03 18:06:14.693597 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:14.693562    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:06:14.693707 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:14.693669    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:06:14.693752 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:14.693718    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:06:14.694483 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:14.694457    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227429c2-kl7ck" event={"ID":"f0e395dc-0519-4588-aa3e-aa9067700361","Type":"ContainerStarted","Data":"b849d7487959cb5330f25373d93b9d894b7071c31f5f391360b0872c896b1e94"}
Nov 03 18:06:14.696020 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:14.695994    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daea96db-846wr" event={"ID":"327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98","Type":"ContainerStarted","Data":"9ee6c8291538ce0eb1c54a8d71f2c4bb8a002e3da0e1360d4e08ca09e15de2f5"}
Nov 03 18:06:14.697499 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:14.697478    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj" event={"ID":"bd9b0713-a9c7-4344-8b32-487d235f9a72","Type":"ContainerStarted","Data":"419b63653f48325ff899a3d1f5a17153275403b2696ce6ebc1e57cec7f63f109"}
Nov 03 18:06:14.887866 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:14.887815    5315 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 03 18:06:14.887995 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:14.887892    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m" podUID="df589ab9-3107-4b55-9195-6ee315bd3965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 03 18:06:15.148017 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:15.147982    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f6488e2-977b-4e72-8467-e242a1aaccee" path="/var/lib/kubelet/pods/5f6488e2-977b-4e72-8467-e242a1aaccee/volumes"
Nov 03 18:06:15.557372 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:15.557338    5315 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 18:06:15.557705 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:15.557672    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs" podUID="1977c777-abc0-4d10-a0ee-3985dd2be5be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 18:06:15.707651 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:15.707594    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp" event={"ID":"59f3868f-6b95-4d02-afe7-4e5228cf37c3","Type":"ContainerStarted","Data":"e9f733f4fd8b9f3f646c0164b8eb5616c00eca87ff3c03c464c9217d024cc71f"}
Nov 03 18:06:15.707825 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:15.707678    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp" event={"ID":"59f3868f-6b95-4d02-afe7-4e5228cf37c3","Type":"ContainerStarted","Data":"c7424cb8369383e8bc1200d6fe00b6f49a5c3bd44a0acead0723bd2511374fa0"}
Nov 03 18:06:15.709448 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:15.709375    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj" event={"ID":"bd9b0713-a9c7-4344-8b32-487d235f9a72","Type":"ContainerStarted","Data":"043eaf103f43bb3b1032c7ba51943da1b159dbd12fd55c4ce69d1f58a2998d23"}
Nov 03 18:06:15.710857 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:15.710836    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227429c2-kl7ck" event={"ID":"f0e395dc-0519-4588-aa3e-aa9067700361","Type":"ContainerStarted","Data":"a1f7c973f11c0bca3a70ceacf0e1dc17ea76b666dd2cca98f3a95f25ec28a551"}
Nov 03 18:06:15.712136 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:15.712111    5315 generic.go:356] "Generic (PLEG): container finished" podID="327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98" containerID="c4f3e8791f7695bc87d8be1a5bc9933b2c479777778756cd32e002233f468dcd" exitCode=0
Nov 03 18:06:15.712232 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:15.712217    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daea96db-846wr" event={"ID":"327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98","Type":"ContainerDied","Data":"c4f3e8791f7695bc87d8be1a5bc9933b2c479777778756cd32e002233f468dcd"}
Nov 03 18:06:15.870278 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:15.870181    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp" podStartSLOduration=2.870157645 podStartE2EDuration="2.870157645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:06:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:06:15.82966175 +0000 UTC m=+1214.845859820" watchObservedRunningTime="2026-11-03 18:06:15.870157645 +0000 UTC m=+1214.886355715"
Nov 03 18:06:15.987340 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:15.906736    5315 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 03 18:06:15.987340 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:15.906818    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m" podUID="df589ab9-3107-4b55-9195-6ee315bd3965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 03 18:06:16.130666 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:16.122044    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj"
Nov 03 18:06:16.130666 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:16.124257    5315 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Nov 03 18:06:16.130666 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:16.124320    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj" podUID="bd9b0713-a9c7-4344-8b32-487d235f9a72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Nov 03 18:06:16.151109 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:16.136154    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj" podStartSLOduration=3.136131101 podStartE2EDuration="3.136131101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:06:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:06:16.124383022 +0000 UTC m=+1215.140581092" watchObservedRunningTime="2026-11-03 18:06:16.136131101 +0000 UTC m=+1215.152329171"
Nov 03 18:06:16.183775 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:16.183737    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp"
Nov 03 18:06:16.309654 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:16.309595    5315 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 03 18:06:16.309936 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:16.309901    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp" podUID="59f3868f-6b95-4d02-afe7-4e5228cf37c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 03 18:06:16.700870 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:16.700778    5315 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 18:06:16.705508 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:16.705458    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs" podUID="1977c777-abc0-4d10-a0ee-3985dd2be5be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 18:06:16.942643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:16.919939    5315 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 03 18:06:16.942643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:16.920000    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m" podUID="df589ab9-3107-4b55-9195-6ee315bd3965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 03 18:06:16.942643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:16.931930    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227429c2-kl7ck" podStartSLOduration=3.931907114 podStartE2EDuration="3.931907114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:06:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:06:16.693320462 +0000 UTC m=+1215.709518532" watchObservedRunningTime="2026-11-03 18:06:16.931907114 +0000 UTC m=+1215.948105185"
Nov 03 18:06:17.031688 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:17.027933    5315 generic.go:356] "Generic (PLEG): container finished" podID="f0e395dc-0519-4588-aa3e-aa9067700361" containerID="a1f7c973f11c0bca3a70ceacf0e1dc17ea76b666dd2cca98f3a95f25ec28a551" exitCode=0
Nov 03 18:06:17.031688 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:17.028836    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227429c2-kl7ck" event={"ID":"f0e395dc-0519-4588-aa3e-aa9067700361","Type":"ContainerDied","Data":"a1f7c973f11c0bca3a70ceacf0e1dc17ea76b666dd2cca98f3a95f25ec28a551"}
Nov 03 18:06:17.153000 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:17.150461    5315 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 03 18:06:17.153000 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:17.150523    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp" podUID="59f3868f-6b95-4d02-afe7-4e5228cf37c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 03 18:06:17.153000 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:17.150859    5315 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Nov 03 18:06:17.153000 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:17.150893    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj" podUID="bd9b0713-a9c7-4344-8b32-487d235f9a72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Nov 03 18:06:17.185783 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:17.185741    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l" podUID="114dcf54-e4eb-4294-a77b-604fe6d57d48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Nov 03 18:06:17.251636 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:17.248820    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6814726-rtxxz"
Nov 03 18:06:17.312889 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:17.303022    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3224261c-8cfa-4f55-90e6-cbbd099f8de7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3224261c-8cfa-4f55-90e6-cbbd099f8de7-maas-api-xks-ca\") pod \"3224261c-8cfa-4f55-90e6-cbbd099f8de7\" (UID: \"3224261c-8cfa-4f55-90e6-cbbd099f8de7\") "
Nov 03 18:06:17.312889 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:17.303185    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3224261c-8cfa-4f55-90e6-cbbd099f8de7-kube-api-access-qq2gd\" (UniqueName: \"kubernetes.io/projected/3224261c-8cfa-4f55-90e6-cbbd099f8de7-kube-api-access-qq2gd\") pod \"3224261c-8cfa-4f55-90e6-cbbd099f8de7\" (UID: \"3224261c-8cfa-4f55-90e6-cbbd099f8de7\") "
Nov 03 18:06:17.312889 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:17.303328    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3224261c-8cfa-4f55-90e6-cbbd099f8de7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3224261c-8cfa-4f55-90e6-cbbd099f8de7-maas-api-openshift-ca\") pod \"3224261c-8cfa-4f55-90e6-cbbd099f8de7\" (UID: \"3224261c-8cfa-4f55-90e6-cbbd099f8de7\") "
Nov 03 18:06:17.313407 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:17.313370    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3224261c-8cfa-4f55-90e6-cbbd099f8de7-maas-api-openshift-ca" pod "3224261c-8cfa-4f55-90e6-cbbd099f8de7" (UID: "3224261c-8cfa-4f55-90e6-cbbd099f8de7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:06:17.315003 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:17.314185    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt" podUID="7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:17.334334 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:17.333960    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv" podUID="f6d706cb-3aa5-419c-8f6c-29347bb788b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:17.334334 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:17.334184    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3224261c-8cfa-4f55-90e6-cbbd099f8de7-maas-api-xks-ca" pod "3224261c-8cfa-4f55-90e6-cbbd099f8de7" (UID: "3224261c-8cfa-4f55-90e6-cbbd099f8de7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:06:17.338937 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:17.338889    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3224261c-8cfa-4f55-90e6-cbbd099f8de7-kube-api-access-qq2gd" pod "3224261c-8cfa-4f55-90e6-cbbd099f8de7" (UID: "3224261c-8cfa-4f55-90e6-cbbd099f8de7"). InnerVolumeSpecName "kube-api-access-qq2gd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:06:17.430771 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:17.430321    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7" podUID="09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Nov 03 18:06:17.534672 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:17.513719    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq2gd\" (UniqueName: \"kubernetes.io/projected/3224261c-8cfa-4f55-90e6-cbbd099f8de7-kube-api-access-qq2gd\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:17.534672 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:17.513760    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3224261c-8cfa-4f55-90e6-cbbd099f8de7-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:17.534672 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:17.513778    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3224261c-8cfa-4f55-90e6-cbbd099f8de7-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:17.563689 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:17.550105    5315 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 18:06:17.563689 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:17.550163    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs" podUID="1977c777-abc0-4d10-a0ee-3985dd2be5be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 18:06:17.742634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:17.712364    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr" podUID="08159a02-fc2e-4138-824d-be5ba3e141d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:17.742634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:17.734911    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q" podUID="c974b0f1-e00b-4e4a-b193-df769068983d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 18:06:17.779655 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:17.718673    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr" podUID="df8e0b71-e876-4cea-a7ae-3532c887c660" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:18.025146 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:17.993456    5315 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 03 18:06:18.025146 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:17.993521    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m" podUID="df589ab9-3107-4b55-9195-6ee315bd3965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 03 18:06:18.067399 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:18.060847    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c" podUID="8ac59539-8ac1-466e-8d1b-b2be4c42c708" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Nov 03 18:06:18.067399 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:18.061695    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s" podUID="155e4b39-1877-4bd9-8541-5d6bd3aa0a8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:18.202636 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:18.161358    5315 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv" podUID="f6d706cb-3aa5-419c-8f6c-29347bb788b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:18.444642 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:18.348440    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv" podUID="f6d706cb-3aa5-419c-8f6c-29347bb788b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 18:06:18.444642 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:18.348863    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt" podUID="7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Nov 03 18:06:18.444642 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:18.349218    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv"
Nov 03 18:06:18.444642 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:18.349282    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt"
Nov 03 18:06:18.464845 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:18.464801    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l" podUID="114dcf54-e4eb-4294-a77b-604fe6d57d48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:18.465474 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:18.465449    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l"
Nov 03 18:06:18.494664 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:18.479889    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7" podUID="09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:18.494664 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:18.480320    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7"
Nov 03 18:06:18.494664 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:18.481282    5315 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Nov 03 18:06:18.494664 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:18.481319    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj" podUID="bd9b0713-a9c7-4344-8b32-487d235f9a72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Nov 03 18:06:18.495723 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:18.495692    5315 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 03 18:06:18.495942 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:18.495917    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp" podUID="59f3868f-6b95-4d02-afe7-4e5228cf37c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 03 18:06:18.568651 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:18.555018    5315 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 18:06:18.568651 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:18.555094    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs" podUID="1977c777-abc0-4d10-a0ee-3985dd2be5be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 18:06:18.716659 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:18.691814    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daea96db-846wr"
Nov 03 18:06:18.737762 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:18.737708    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr" podUID="08159a02-fc2e-4138-824d-be5ba3e141d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:18.773636 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:18.748078    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr"
Nov 03 18:06:18.774234 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:18.774197    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6814726-rtxxz" event={"ID":"3224261c-8cfa-4f55-90e6-cbbd099f8de7","Type":"ContainerDied","Data":"2990983aa101106f7209be1e2d6bcfc5a0edb61aca20edaf9f815f3f6dd0e6b2"}
Nov 03 18:06:18.778834 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:18.778800    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2990983aa101106f7209be1e2d6bcfc5a0edb61aca20edaf9f815f3f6dd0e6b2"
Nov 03 18:06:18.778985 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:18.759185    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6814726-rtxxz"
Nov 03 18:06:18.807675 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:18.760432    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q" podUID="c974b0f1-e00b-4e4a-b193-df769068983d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:18.808469 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:18.808444    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q"
Nov 03 18:06:18.857220 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:18.857116    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98-kube-api-access-7r6w4\" (UniqueName: \"kubernetes.io/projected/327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98-kube-api-access-7r6w4\") pod \"327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98\" (UID: \"327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98\") "
Nov 03 18:06:18.857600 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:18.857575    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98-maas-api-openshift-ca\") pod \"327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98\" (UID: \"327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98\") "
Nov 03 18:06:18.857798 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:18.857779    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98-maas-api-xks-ca\") pod \"327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98\" (UID: \"327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98\") "
Nov 03 18:06:18.914537 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:18.914496    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98-maas-api-openshift-ca" pod "327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98" (UID: "327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:06:18.996654 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:18.985837    5315 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 03 18:06:18.996654 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:18.985905    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m" podUID="df589ab9-3107-4b55-9195-6ee315bd3965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 03 18:06:18.996654 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:18.986711    5315 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7" podUID="09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": context deadline exceeded"
Nov 03 18:06:18.996654 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:18.987328    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr" podUID="df8e0b71-e876-4cea-a7ae-3532c887c660" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:18.996654 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:18.988263    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr"
Nov 03 18:06:19.098634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:19.098055    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7" podUID="69760dba-faf9-4c08-895c-ad70fe95fd49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:19.161931 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:19.126715    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c" podUID="8ac59539-8ac1-466e-8d1b-b2be4c42c708" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Nov 03 18:06:19.161931 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:19.151746    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s" podUID="155e4b39-1877-4bd9-8541-5d6bd3aa0a8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Nov 03 18:06:19.167559 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:19.162829    5315 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 03 18:06:19.167559 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:19.162888    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp" podUID="59f3868f-6b95-4d02-afe7-4e5228cf37c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 03 18:06:19.259836 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:19.204403    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7" podUID="33725a51-6459-4f15-a866-e9dee41a9732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 03 18:06:19.259836 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:19.241802    5315 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Nov 03 18:06:19.259836 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:19.241861    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj" podUID="bd9b0713-a9c7-4344-8b32-487d235f9a72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Nov 03 18:06:19.384672 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:19.348015    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:19.435693 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:19.387648    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv" podUID="f6d706cb-3aa5-419c-8f6c-29347bb788b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 18:06:19.466266 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:19.439751    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt" podUID="7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Nov 03 18:06:19.644849 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:19.631764    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l" podUID="114dcf54-e4eb-4294-a77b-604fe6d57d48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Nov 03 18:06:19.708701 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:19.685395    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98-maas-api-xks-ca" pod "327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98" (UID: "327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:06:19.708883 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:19.708792    5315 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 18:06:19.708883 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:19.708856    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs" podUID="1977c777-abc0-4d10-a0ee-3985dd2be5be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 18:06:19.724941 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:19.724883    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7" podUID="09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Nov 03 18:06:19.803631 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:19.796408    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr" podUID="08159a02-fc2e-4138-824d-be5ba3e141d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:19.896030 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:19.805929    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98-kube-api-access-7r6w4" pod "327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98" (UID: "327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98"). InnerVolumeSpecName "kube-api-access-7r6w4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:06:19.995752 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:19.943747    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q" podUID="c974b0f1-e00b-4e4a-b193-df769068983d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:20.022298 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:20.014603    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:20.289899 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:20.289847    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r6w4\" (UniqueName: \"kubernetes.io/projected/327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98-kube-api-access-7r6w4\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:20.290469 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:20.100482    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c"
Nov 03 18:06:20.290944 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:20.290911    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s"
Nov 03 18:06:20.305196 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:20.129894    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c" podUID="8ac59539-8ac1-466e-8d1b-b2be4c42c708" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:20.305775 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:20.156019    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr" podUID="df8e0b71-e876-4cea-a7ae-3532c887c660" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:20.306558 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:20.168329    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s" podUID="155e4b39-1877-4bd9-8541-5d6bd3aa0a8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:20.306963 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:20.175737    5315 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 03 18:06:20.307118 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:20.307090    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m" podUID="df589ab9-3107-4b55-9195-6ee315bd3965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 03 18:06:20.307246 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:20.019380    5315 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt" podUID="7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:20.307368 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:20.113809    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7" podUID="69760dba-faf9-4c08-895c-ad70fe95fd49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:20.307767 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:20.214175    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7" podUID="33725a51-6459-4f15-a866-e9dee41a9732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:20.335724 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:20.335666    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7"
Nov 03 18:06:20.346970 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:20.346939    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7"
Nov 03 18:06:20.347692 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:20.176808    5315 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 03 18:06:20.480807 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:20.177073    5315 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Nov 03 18:06:20.481159 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:20.481124    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj" podUID="bd9b0713-a9c7-4344-8b32-487d235f9a72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Nov 03 18:06:20.497798 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:20.478119    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv" podUID="f6d706cb-3aa5-419c-8f6c-29347bb788b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:20.498426 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:20.481016    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp" podUID="59f3868f-6b95-4d02-afe7-4e5228cf37c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 03 18:06:20.498639 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:20.477802    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt" podUID="7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:20.503207 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:20.503174    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227429c2-kl7ck" event={"ID":"f0e395dc-0519-4588-aa3e-aa9067700361","Type":"ContainerDied","Data":"b849d7487959cb5330f25373d93b9d894b7071c31f5f391360b0872c896b1e94"}
Nov 03 18:06:20.503417 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:20.503396    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b849d7487959cb5330f25373d93b9d894b7071c31f5f391360b0872c896b1e94"
Nov 03 18:06:20.511670 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:20.506938    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227429c2-kl7ck"
Nov 03 18:06:20.546735 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:20.538584    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daea96db-846wr" event={"ID":"327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98","Type":"ContainerDied","Data":"9ee6c8291538ce0eb1c54a8d71f2c4bb8a002e3da0e1360d4e08ca09e15de2f5"}
Nov 03 18:06:20.573889 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:20.573848    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ee6c8291538ce0eb1c54a8d71f2c4bb8a002e3da0e1360d4e08ca09e15de2f5"
Nov 03 18:06:20.574133 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:20.547178    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daea96db-846wr"
Nov 03 18:06:20.633478 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:20.633436    5315 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 18:06:20.633920 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:20.633875    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l" podUID="114dcf54-e4eb-4294-a77b-604fe6d57d48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:20.634393 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:20.633802    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs" podUID="1977c777-abc0-4d10-a0ee-3985dd2be5be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 18:06:20.695650 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:20.694016    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0e395dc-0519-4588-aa3e-aa9067700361-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0e395dc-0519-4588-aa3e-aa9067700361-maas-api-openshift-ca\") pod \"f0e395dc-0519-4588-aa3e-aa9067700361\" (UID: \"f0e395dc-0519-4588-aa3e-aa9067700361\") "
Nov 03 18:06:20.695650 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:20.694298    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0e395dc-0519-4588-aa3e-aa9067700361-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0e395dc-0519-4588-aa3e-aa9067700361-maas-api-xks-ca\") pod \"f0e395dc-0519-4588-aa3e-aa9067700361\" (UID: \"f0e395dc-0519-4588-aa3e-aa9067700361\") "
Nov 03 18:06:20.695650 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:20.694443    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0e395dc-0519-4588-aa3e-aa9067700361-kube-api-access-9wfzf\" (UniqueName: \"kubernetes.io/projected/f0e395dc-0519-4588-aa3e-aa9067700361-kube-api-access-9wfzf\") pod \"f0e395dc-0519-4588-aa3e-aa9067700361\" (UID: \"f0e395dc-0519-4588-aa3e-aa9067700361\") "
Nov 03 18:06:20.695947 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:20.695690    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0e395dc-0519-4588-aa3e-aa9067700361-maas-api-openshift-ca" pod "f0e395dc-0519-4588-aa3e-aa9067700361" (UID: "f0e395dc-0519-4588-aa3e-aa9067700361"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:06:20.706466 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:20.706433    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0e395dc-0519-4588-aa3e-aa9067700361-kube-api-access-9wfzf" pod "f0e395dc-0519-4588-aa3e-aa9067700361" (UID: "f0e395dc-0519-4588-aa3e-aa9067700361"). InnerVolumeSpecName "kube-api-access-9wfzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:06:20.706955 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:20.706932    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0e395dc-0519-4588-aa3e-aa9067700361-maas-api-xks-ca" pod "f0e395dc-0519-4588-aa3e-aa9067700361" (UID: "f0e395dc-0519-4588-aa3e-aa9067700361"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:06:20.779635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:20.755785    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7" podUID="09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:20.809295 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:20.809192    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr" podUID="08159a02-fc2e-4138-824d-be5ba3e141d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:20.922823 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:20.922758    5315 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 03 18:06:20.923168 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:20.923137    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m" podUID="df589ab9-3107-4b55-9195-6ee315bd3965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 03 18:06:20.966850 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:20.966814    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0e395dc-0519-4588-aa3e-aa9067700361-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:20.967064 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:20.967044    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wfzf\" (UniqueName: \"kubernetes.io/projected/f0e395dc-0519-4588-aa3e-aa9067700361-kube-api-access-9wfzf\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:20.967187 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:20.967174    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0e395dc-0519-4588-aa3e-aa9067700361-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:20.988859 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:20.988817    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q" podUID="c974b0f1-e00b-4e4a-b193-df769068983d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 18:06:21.181723 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:21.181077    5315 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Nov 03 18:06:21.181723 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:21.181247    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj" podUID="bd9b0713-a9c7-4344-8b32-487d235f9a72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Nov 03 18:06:21.212429 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:21.212380    5315 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 03 18:06:21.214999 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:21.214964    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp" podUID="59f3868f-6b95-4d02-afe7-4e5228cf37c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 03 18:06:21.313666 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:21.312782    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7" podUID="69760dba-faf9-4c08-895c-ad70fe95fd49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/ready\": context deadline exceeded"
Nov 03 18:06:21.314130 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:21.313860    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr" podUID="df8e0b71-e876-4cea-a7ae-3532c887c660" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 18:06:21.314130 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:21.313938    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s" podUID="155e4b39-1877-4bd9-8541-5d6bd3aa0a8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Nov 03 18:06:21.378657 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:21.363690    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7" podUID="33725a51-6459-4f15-a866-e9dee41a9732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:21.501450 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:21.501029    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt" podUID="7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:21.502702 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:21.502673    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c"
Nov 03 18:06:21.510790 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:21.510240    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l"
Nov 03 18:06:21.510790 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:21.510286    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d0be55c-rzk6z"]
Nov 03 18:06:21.510790 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:21.510343    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv"
Nov 03 18:06:21.510790 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:21.510367    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d0be55c-rzk6z"]
Nov 03 18:06:21.537659 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:21.525144    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7"
Nov 03 18:06:21.575634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:21.571740    5315 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 18:06:21.575634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:21.571812    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs" podUID="1977c777-abc0-4d10-a0ee-3985dd2be5be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 18:06:21.679640 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:21.671751    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s"
Nov 03 18:06:21.715815 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:21.713653    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr"
Nov 03 18:06:21.749632 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:21.749154    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227429c2-kl7ck"
Nov 03 18:06:21.810639 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:21.810538    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr" podUID="08159a02-fc2e-4138-824d-be5ba3e141d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:21.999408 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:21.999371    5315 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 03 18:06:21.999672 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:21.999640    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m" podUID="df589ab9-3107-4b55-9195-6ee315bd3965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 03 18:06:22.002215 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:22.002181    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q" podUID="c974b0f1-e00b-4e4a-b193-df769068983d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:22.009635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:22.005227    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr"
Nov 03 18:06:22.018232 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:22.018206    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt"
Nov 03 18:06:22.020835 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:22.020814    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q"
Nov 03 18:06:22.093833 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:22.093740    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f88012a-7nkk7"]
Nov 03 18:06:22.100476 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:22.100439    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f88012a-7nkk7"]
Nov 03 18:06:22.168708 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:22.168675    5315 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Nov 03 18:06:22.180766 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:22.180720    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj" podUID="bd9b0713-a9c7-4344-8b32-487d235f9a72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Nov 03 18:06:22.181015 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:22.168958    5315 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 03 18:06:22.181145 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:22.181124    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp" podUID="59f3868f-6b95-4d02-afe7-4e5228cf37c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 03 18:06:22.503696 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:22.478377    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7" podUID="69760dba-faf9-4c08-895c-ad70fe95fd49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:22.505042 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:22.500473    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7" podUID="33725a51-6459-4f15-a866-e9dee41a9732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:22.998730 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:22.998418    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7"
Nov 03 18:06:23.081628 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:23.075968    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m"
Nov 03 18:06:23.081628 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:23.077310    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m"
Nov 03 18:06:23.081628 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:23.077528    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7"
Nov 03 18:06:23.081910 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:23.081672    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m"
Nov 03 18:06:23.121231 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:23.121196    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94c7f5f6-0b51-48a0-b7b6-81edc7bff526" path="/var/lib/kubelet/pods/94c7f5f6-0b51-48a0-b7b6-81edc7bff526/volumes"
Nov 03 18:06:23.122528 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:23.122497    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a12e0e65-73d8-4441-873f-df3978fc0a04" path="/var/lib/kubelet/pods/a12e0e65-73d8-4441-873f-df3978fc0a04/volumes"
Nov 03 18:06:23.131428 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:23.128158    5315 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Nov 03 18:06:23.131428 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:23.128232    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj" podUID="bd9b0713-a9c7-4344-8b32-487d235f9a72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Nov 03 18:06:23.136372 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:23.136342    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs"
Nov 03 18:06:23.137624 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:23.137584    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs"
Nov 03 18:06:23.139277 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:23.138470    5315 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 03 18:06:23.139277 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:23.138517    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp" podUID="59f3868f-6b95-4d02-afe7-4e5228cf37c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 03 18:06:23.144843 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:23.144814    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs"
Nov 03 18:06:24.398522 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:24.390972    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp"
Nov 03 18:06:24.398522 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:24.392836    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp"
Nov 03 18:06:24.400653 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:24.399928    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp"
Nov 03 18:06:24.414501 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:24.412957    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80bee53a-zmp7n"]
Nov 03 18:06:24.443402 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:24.441317    5315 cpu_manager.go:414] "containerMap: removing container" podUID="f0e395dc-0519-4588-aa3e-aa9067700361" containerName="revoke-keys"
Nov 03 18:06:24.443402 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:24.441365    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0e395dc-0519-4588-aa3e-aa9067700361" containerName="revoke-keys"
Nov 03 18:06:24.443402 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:24.441385    5315 cpu_manager.go:414] "containerMap: removing container" podUID="3224261c-8cfa-4f55-90e6-cbbd099f8de7" containerName="revoke-keys"
Nov 03 18:06:24.443402 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:24.441393    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3224261c-8cfa-4f55-90e6-cbbd099f8de7" containerName="revoke-keys"
Nov 03 18:06:24.443402 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:24.441434    5315 cpu_manager.go:414] "containerMap: removing container" podUID="327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98" containerName="revoke-keys"
Nov 03 18:06:24.443402 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:24.441443    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98" containerName="revoke-keys"
Nov 03 18:06:24.443402 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:24.441670    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98" containerName="revoke-keys"
Nov 03 18:06:24.443402 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:24.441711    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0e395dc-0519-4588-aa3e-aa9067700361" containerName="revoke-keys"
Nov 03 18:06:24.443402 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:24.441724    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="3224261c-8cfa-4f55-90e6-cbbd099f8de7" containerName="revoke-keys"
Nov 03 18:06:24.459367 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:24.456303    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80bee53a-zmp7n"
Nov 03 18:06:24.474215 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:24.474177    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80bee53a-zmp7n"]
Nov 03 18:06:24.484666 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:24.483356    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj"
Nov 03 18:06:24.486022 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:24.485980    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj"
Nov 03 18:06:24.532087 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:24.503629    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzrjh\" (UniqueName: \"kubernetes.io/projected/2458d282-5d5f-4967-8536-85ba89c9eec3-kube-api-access-xzrjh\") pod \"maas-api-revoke-sub-80bee53a-zmp7n\" (UID: \"2458d282-5d5f-4967-8536-85ba89c9eec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80bee53a-zmp7n"
Nov 03 18:06:24.532087 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:24.503797    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2458d282-5d5f-4967-8536-85ba89c9eec3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80bee53a-zmp7n\" (UID: \"2458d282-5d5f-4967-8536-85ba89c9eec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80bee53a-zmp7n"
Nov 03 18:06:24.532087 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:24.503841    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2458d282-5d5f-4967-8536-85ba89c9eec3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80bee53a-zmp7n\" (UID: \"2458d282-5d5f-4967-8536-85ba89c9eec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80bee53a-zmp7n"
Nov 03 18:06:24.532087 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:24.511353    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:06:24.532087 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:24.519379    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:06:24.532087 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:24.519447    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:06:24.532087 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:24.519521    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:06:24.536625 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:24.536141    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj"
Nov 03 18:06:24.578224 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:06:24.578168    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:06:24.605152 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:24.605119    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzrjh\" (UniqueName: \"kubernetes.io/projected/2458d282-5d5f-4967-8536-85ba89c9eec3-kube-api-access-xzrjh\") pod \"maas-api-revoke-sub-80bee53a-zmp7n\" (UID: \"2458d282-5d5f-4967-8536-85ba89c9eec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80bee53a-zmp7n"
Nov 03 18:06:24.605318 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:24.605226    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2458d282-5d5f-4967-8536-85ba89c9eec3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80bee53a-zmp7n\" (UID: \"2458d282-5d5f-4967-8536-85ba89c9eec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80bee53a-zmp7n"
Nov 03 18:06:24.605318 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:24.605267    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2458d282-5d5f-4967-8536-85ba89c9eec3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80bee53a-zmp7n\" (UID: \"2458d282-5d5f-4967-8536-85ba89c9eec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80bee53a-zmp7n"
Nov 03 18:06:24.605894 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:24.605868    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2458d282-5d5f-4967-8536-85ba89c9eec3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80bee53a-zmp7n\" (UID: \"2458d282-5d5f-4967-8536-85ba89c9eec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80bee53a-zmp7n"
Nov 03 18:06:24.607531 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:24.607510    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2458d282-5d5f-4967-8536-85ba89c9eec3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80bee53a-zmp7n\" (UID: \"2458d282-5d5f-4967-8536-85ba89c9eec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80bee53a-zmp7n"
Nov 03 18:06:24.662820 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:24.662742    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzrjh\" (UniqueName: \"kubernetes.io/projected/2458d282-5d5f-4967-8536-85ba89c9eec3-kube-api-access-xzrjh\") pod \"maas-api-revoke-sub-80bee53a-zmp7n\" (UID: \"2458d282-5d5f-4967-8536-85ba89c9eec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80bee53a-zmp7n"
Nov 03 18:06:24.842058 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:24.842022    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80bee53a-zmp7n"
Nov 03 18:06:25.057545 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:25.057504    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e91a8237-vvgc5"]
Nov 03 18:06:25.067827 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:25.067790    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e91a8237-vvgc5"]
Nov 03 18:06:25.106339 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:25.106298    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2943536-963c-462e-9715-7756e5af31f6" path="/var/lib/kubelet/pods/f2943536-963c-462e-9715-7756e5af31f6/volumes"
Nov 03 18:06:25.220078 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:25.220041    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80bee53a-zmp7n"]
Nov 03 18:06:25.225048 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:06:25.225005    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2458d282_5d5f_4967_8536_85ba89c9eec3.slice/crio-9c36a77e883a403611482be3f31a7f056d911246ceeb46a8582a3ad240d70549 WatchSource:0}: Error finding container 9c36a77e883a403611482be3f31a7f056d911246ceeb46a8582a3ad240d70549: Status 404 returned error can't find the container with id 9c36a77e883a403611482be3f31a7f056d911246ceeb46a8582a3ad240d70549
Nov 03 18:06:25.477561 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:25.477531    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7353-c12ad0"]
Nov 03 18:06:25.483295 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:25.483274    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7353-c12ad0"
Nov 03 18:06:25.520133 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:25.520097    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqznk\" (UniqueName: \"kubernetes.io/projected/e34e974c-1009-4a89-9735-2493179d9411-kube-api-access-jqznk\") pod \"test-curl-7353-c12ad0\" (UID: \"e34e974c-1009-4a89-9735-2493179d9411\") " pod="opendatahub/test-curl-7353-c12ad0"
Nov 03 18:06:25.531955 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:25.531900    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-447v9\""
Nov 03 18:06:25.538662 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:25.538639    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7353-c12ad0"]
Nov 03 18:06:25.621548 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:25.621511    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqznk\" (UniqueName: \"kubernetes.io/projected/e34e974c-1009-4a89-9735-2493179d9411-kube-api-access-jqznk\") pod \"test-curl-7353-c12ad0\" (UID: \"e34e974c-1009-4a89-9735-2493179d9411\") " pod="opendatahub/test-curl-7353-c12ad0"
Nov 03 18:06:25.678453 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:25.678421    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqznk\" (UniqueName: \"kubernetes.io/projected/e34e974c-1009-4a89-9735-2493179d9411-kube-api-access-jqznk\") pod \"test-curl-7353-c12ad0\" (UID: \"e34e974c-1009-4a89-9735-2493179d9411\") " pod="opendatahub/test-curl-7353-c12ad0"
Nov 03 18:06:25.753450 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:25.753360    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4298e49-xjwqp"]
Nov 03 18:06:25.760271 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:25.760245    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4298e49-xjwqp"
Nov 03 18:06:25.795712 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:25.795679    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7353-c12ad0"
Nov 03 18:06:25.820897 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:25.820864    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4298e49-xjwqp"]
Nov 03 18:06:25.824974 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:25.824471    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkg9h\" (UniqueName: \"kubernetes.io/projected/518883f2-c175-4ad7-a4d0-15dfa5520315-kube-api-access-vkg9h\") pod \"maas-api-revoke-sub-c4298e49-xjwqp\" (UID: \"518883f2-c175-4ad7-a4d0-15dfa5520315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4298e49-xjwqp"
Nov 03 18:06:25.824974 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:25.824733    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/518883f2-c175-4ad7-a4d0-15dfa5520315-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4298e49-xjwqp\" (UID: \"518883f2-c175-4ad7-a4d0-15dfa5520315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4298e49-xjwqp"
Nov 03 18:06:25.824974 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:25.824845    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/518883f2-c175-4ad7-a4d0-15dfa5520315-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4298e49-xjwqp\" (UID: \"518883f2-c175-4ad7-a4d0-15dfa5520315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4298e49-xjwqp"
Nov 03 18:06:25.888854 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:25.884802    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80bee53a-zmp7n" event={"ID":"2458d282-5d5f-4967-8536-85ba89c9eec3","Type":"ContainerStarted","Data":"435ed6f4608146f2941f806a756f18a9a6b48e00b313ae574c631f858d7d657d"}
Nov 03 18:06:25.888854 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:25.884853    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80bee53a-zmp7n" event={"ID":"2458d282-5d5f-4967-8536-85ba89c9eec3","Type":"ContainerStarted","Data":"9c36a77e883a403611482be3f31a7f056d911246ceeb46a8582a3ad240d70549"}
Nov 03 18:06:25.928243 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:25.926470    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/518883f2-c175-4ad7-a4d0-15dfa5520315-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4298e49-xjwqp\" (UID: \"518883f2-c175-4ad7-a4d0-15dfa5520315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4298e49-xjwqp"
Nov 03 18:06:25.928243 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:25.926657    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/518883f2-c175-4ad7-a4d0-15dfa5520315-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4298e49-xjwqp\" (UID: \"518883f2-c175-4ad7-a4d0-15dfa5520315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4298e49-xjwqp"
Nov 03 18:06:25.928243 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:25.926707    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkg9h\" (UniqueName: \"kubernetes.io/projected/518883f2-c175-4ad7-a4d0-15dfa5520315-kube-api-access-vkg9h\") pod \"maas-api-revoke-sub-c4298e49-xjwqp\" (UID: \"518883f2-c175-4ad7-a4d0-15dfa5520315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4298e49-xjwqp"
Nov 03 18:06:25.928243 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:25.927926    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/518883f2-c175-4ad7-a4d0-15dfa5520315-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4298e49-xjwqp\" (UID: \"518883f2-c175-4ad7-a4d0-15dfa5520315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4298e49-xjwqp"
Nov 03 18:06:25.930953 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:25.930923    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/518883f2-c175-4ad7-a4d0-15dfa5520315-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4298e49-xjwqp\" (UID: \"518883f2-c175-4ad7-a4d0-15dfa5520315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4298e49-xjwqp"
Nov 03 18:06:25.985071 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:25.985035    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkg9h\" (UniqueName: \"kubernetes.io/projected/518883f2-c175-4ad7-a4d0-15dfa5520315-kube-api-access-vkg9h\") pod \"maas-api-revoke-sub-c4298e49-xjwqp\" (UID: \"518883f2-c175-4ad7-a4d0-15dfa5520315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4298e49-xjwqp"
Nov 03 18:06:25.997198 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:25.997150    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80bee53a-zmp7n" podStartSLOduration=1.9971361939999999 podStartE2EDuration="1.997136194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:06:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:06:25.994831699 +0000 UTC m=+1225.011029774" watchObservedRunningTime="2026-11-03 18:06:25.997136194 +0000 UTC m=+1225.013334273"
Nov 03 18:06:26.048798 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:26.048759    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00110e0a-hj978"]
Nov 03 18:06:26.052938 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:26.052911    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00110e0a-hj978"]
Nov 03 18:06:26.080432 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:26.080394    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4298e49-xjwqp"
Nov 03 18:06:26.182799 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:26.182729    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7353-c12ad0"]
Nov 03 18:06:26.187584 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:06:26.187557    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode34e974c_1009_4a89_9735_2493179d9411.slice/crio-46a632d36d512824917f9830d46c22bf3cd792e6a08f770e2557817331370188 WatchSource:0}: Error finding container 46a632d36d512824917f9830d46c22bf3cd792e6a08f770e2557817331370188: Status 404 returned error can't find the container with id 46a632d36d512824917f9830d46c22bf3cd792e6a08f770e2557817331370188
Nov 03 18:06:26.189861 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:26.189838    5315 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 18:06:26.459367 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:26.459257    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4298e49-xjwqp"]
Nov 03 18:06:26.463517 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:06:26.463484    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod518883f2_c175_4ad7_a4d0_15dfa5520315.slice/crio-a4d62659a7583e34e3d96dfa67e4bf4ae39508c9dc8b8c0a289c5ed03aa839ec WatchSource:0}: Error finding container a4d62659a7583e34e3d96dfa67e4bf4ae39508c9dc8b8c0a289c5ed03aa839ec: Status 404 returned error can't find the container with id a4d62659a7583e34e3d96dfa67e4bf4ae39508c9dc8b8c0a289c5ed03aa839ec
Nov 03 18:06:26.740214 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:26.740131    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4"]
Nov 03 18:06:26.746440 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:26.746415    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4"
Nov 03 18:06:26.818824 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:26.818793    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4"]
Nov 03 18:06:26.840281 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:26.840248    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-workload-socket\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4\" (UID: \"c3b8cf40-02df-499d-b41c-6e4bd9283a3b\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4"
Nov 03 18:06:26.840421 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:26.840332    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-istio-podinfo\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4\" (UID: \"c3b8cf40-02df-499d-b41c-6e4bd9283a3b\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4"
Nov 03 18:06:26.840550 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:26.840523    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-istio-envoy\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4\" (UID: \"c3b8cf40-02df-499d-b41c-6e4bd9283a3b\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4"
Nov 03 18:06:26.840658 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:26.840644    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-istiod-ca-cert\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4\" (UID: \"c3b8cf40-02df-499d-b41c-6e4bd9283a3b\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4"
Nov 03 18:06:26.840741 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:26.840725    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-credential-socket\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4\" (UID: \"c3b8cf40-02df-499d-b41c-6e4bd9283a3b\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4"
Nov 03 18:06:26.840792 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:26.840771    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk2mm\" (UniqueName: \"kubernetes.io/projected/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-kube-api-access-hk2mm\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4\" (UID: \"c3b8cf40-02df-499d-b41c-6e4bd9283a3b\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4"
Nov 03 18:06:26.840864 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:26.840809    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-istio-token\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4\" (UID: \"c3b8cf40-02df-499d-b41c-6e4bd9283a3b\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4"
Nov 03 18:06:26.841005 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:26.840843    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-workload-certs\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4\" (UID: \"c3b8cf40-02df-499d-b41c-6e4bd9283a3b\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4"
Nov 03 18:06:26.841077 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:26.841062    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-istio-data\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4\" (UID: \"c3b8cf40-02df-499d-b41c-6e4bd9283a3b\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4"
Nov 03 18:06:26.895953 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:26.895915    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4298e49-xjwqp" event={"ID":"518883f2-c175-4ad7-a4d0-15dfa5520315","Type":"ContainerStarted","Data":"33401f9d2ff4ef361cef300206bff2787c0ba7184b7db672c584edf9f2397b71"}
Nov 03 18:06:26.896107 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:26.895983    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4298e49-xjwqp" event={"ID":"518883f2-c175-4ad7-a4d0-15dfa5520315","Type":"ContainerStarted","Data":"a4d62659a7583e34e3d96dfa67e4bf4ae39508c9dc8b8c0a289c5ed03aa839ec"}
Nov 03 18:06:26.897368 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:26.897334    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7353-c12ad0" event={"ID":"e34e974c-1009-4a89-9735-2493179d9411","Type":"ContainerStarted","Data":"46a632d36d512824917f9830d46c22bf3cd792e6a08f770e2557817331370188"}
Nov 03 18:06:26.898873 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:26.898846    5315 generic.go:356] "Generic (PLEG): container finished" podID="2458d282-5d5f-4967-8536-85ba89c9eec3" containerID="435ed6f4608146f2941f806a756f18a9a6b48e00b313ae574c631f858d7d657d" exitCode=0
Nov 03 18:06:26.898982 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:26.898927    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80bee53a-zmp7n" event={"ID":"2458d282-5d5f-4967-8536-85ba89c9eec3","Type":"ContainerDied","Data":"435ed6f4608146f2941f806a756f18a9a6b48e00b313ae574c631f858d7d657d"}
Nov 03 18:06:26.941848 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:26.941815    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-workload-certs\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4\" (UID: \"c3b8cf40-02df-499d-b41c-6e4bd9283a3b\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4"
Nov 03 18:06:26.941992 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:26.941872    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-istio-data\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4\" (UID: \"c3b8cf40-02df-499d-b41c-6e4bd9283a3b\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4"
Nov 03 18:06:26.941992 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:26.941919    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-workload-socket\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4\" (UID: \"c3b8cf40-02df-499d-b41c-6e4bd9283a3b\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4"
Nov 03 18:06:26.941992 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:26.941975    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-istio-podinfo\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4\" (UID: \"c3b8cf40-02df-499d-b41c-6e4bd9283a3b\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4"
Nov 03 18:06:26.942164 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:26.942145    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-istio-envoy\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4\" (UID: \"c3b8cf40-02df-499d-b41c-6e4bd9283a3b\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4"
Nov 03 18:06:26.942243 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:26.942221    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-istiod-ca-cert\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4\" (UID: \"c3b8cf40-02df-499d-b41c-6e4bd9283a3b\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4"
Nov 03 18:06:26.942243 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:26.942226    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-workload-certs\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4\" (UID: \"c3b8cf40-02df-499d-b41c-6e4bd9283a3b\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4"
Nov 03 18:06:26.942353 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:26.942299    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-credential-socket\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4\" (UID: \"c3b8cf40-02df-499d-b41c-6e4bd9283a3b\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4"
Nov 03 18:06:26.942353 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:26.942302    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-workload-socket\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4\" (UID: \"c3b8cf40-02df-499d-b41c-6e4bd9283a3b\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4"
Nov 03 18:06:26.942455 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:26.942354    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-istio-data\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4\" (UID: \"c3b8cf40-02df-499d-b41c-6e4bd9283a3b\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4"
Nov 03 18:06:26.942455 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:26.942359    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk2mm\" (UniqueName: \"kubernetes.io/projected/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-kube-api-access-hk2mm\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4\" (UID: \"c3b8cf40-02df-499d-b41c-6e4bd9283a3b\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4"
Nov 03 18:06:26.942455 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:26.942416    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-istio-token\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4\" (UID: \"c3b8cf40-02df-499d-b41c-6e4bd9283a3b\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4"
Nov 03 18:06:26.942651 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:26.942594    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-credential-socket\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4\" (UID: \"c3b8cf40-02df-499d-b41c-6e4bd9283a3b\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4"
Nov 03 18:06:26.943001 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:26.942979    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-istiod-ca-cert\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4\" (UID: \"c3b8cf40-02df-499d-b41c-6e4bd9283a3b\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4"
Nov 03 18:06:26.944570 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:26.944544    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-istio-envoy\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4\" (UID: \"c3b8cf40-02df-499d-b41c-6e4bd9283a3b\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4"
Nov 03 18:06:26.944776 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:26.944758    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-istio-podinfo\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4\" (UID: \"c3b8cf40-02df-499d-b41c-6e4bd9283a3b\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4"
Nov 03 18:06:27.000511 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:27.000392    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk2mm\" (UniqueName: \"kubernetes.io/projected/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-kube-api-access-hk2mm\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4\" (UID: \"c3b8cf40-02df-499d-b41c-6e4bd9283a3b\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4"
Nov 03 18:06:27.008142 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:27.004217    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-istio-token\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4\" (UID: \"c3b8cf40-02df-499d-b41c-6e4bd9283a3b\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4"
Nov 03 18:06:27.009343 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:27.009289    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4298e49-xjwqp" podStartSLOduration=2.009274485 podStartE2EDuration="2.009274485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:06:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:06:27.003975269 +0000 UTC m=+1226.020173350" watchObservedRunningTime="2026-11-03 18:06:27.009274485 +0000 UTC m=+1226.025472556"
Nov 03 18:06:27.061558 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:27.061526    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4"
Nov 03 18:06:27.110197 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:27.110157    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="581376ba-0009-4555-8999-ba410f3eb7e6" path="/var/lib/kubelet/pods/581376ba-0009-4555-8999-ba410f3eb7e6/volumes"
Nov 03 18:06:27.512587 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:27.512556    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4"]
Nov 03 18:06:27.514735 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:06:27.514704    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3b8cf40_02df_499d_b41c_6e4bd9283a3b.slice/crio-4a864df2a434a7025ec9c065163ab233458937fc1c261d6c389988a83c3e2395 WatchSource:0}: Error finding container 4a864df2a434a7025ec9c065163ab233458937fc1c261d6c389988a83c3e2395: Status 404 returned error can't find the container with id 4a864df2a434a7025ec9c065163ab233458937fc1c261d6c389988a83c3e2395
Nov 03 18:06:27.517743 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:27.517706    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:06:27.517837 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:27.517791    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:06:27.517895 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:27.517836    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:06:27.911603 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:27.911557    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4" event={"ID":"c3b8cf40-02df-499d-b41c-6e4bd9283a3b","Type":"ContainerStarted","Data":"94994b1046d894c179ec978ed84492241f9b524e40aaf36f07a96d48c0bbcf72"}
Nov 03 18:06:27.912168 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:27.911670    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4" event={"ID":"c3b8cf40-02df-499d-b41c-6e4bd9283a3b","Type":"ContainerStarted","Data":"4a864df2a434a7025ec9c065163ab233458937fc1c261d6c389988a83c3e2395"}
Nov 03 18:06:27.914850 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:27.914822    5315 generic.go:356] "Generic (PLEG): container finished" podID="518883f2-c175-4ad7-a4d0-15dfa5520315" containerID="33401f9d2ff4ef361cef300206bff2787c0ba7184b7db672c584edf9f2397b71" exitCode=0
Nov 03 18:06:27.915169 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:27.915141    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4298e49-xjwqp" event={"ID":"518883f2-c175-4ad7-a4d0-15dfa5520315","Type":"ContainerDied","Data":"33401f9d2ff4ef361cef300206bff2787c0ba7184b7db672c584edf9f2397b71"}
Nov 03 18:06:28.034430 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:28.032589    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4" podStartSLOduration=2.032568031 podStartE2EDuration="2.032568031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:06:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:06:28.02796743 +0000 UTC m=+1227.044165500" watchObservedRunningTime="2026-11-03 18:06:28.032568031 +0000 UTC m=+1227.048766102"
Nov 03 18:06:28.061992 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:28.061737    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4"
Nov 03 18:06:28.064918 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:28.064866    5315 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 03 18:06:28.065118 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:28.064929    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4" podUID="c3b8cf40-02df-499d-b41c-6e4bd9283a3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 03 18:06:28.119548 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:28.119518    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80bee53a-zmp7n"
Nov 03 18:06:28.167987 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:28.167905    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2458d282-5d5f-4967-8536-85ba89c9eec3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2458d282-5d5f-4967-8536-85ba89c9eec3-maas-api-xks-ca\") pod \"2458d282-5d5f-4967-8536-85ba89c9eec3\" (UID: \"2458d282-5d5f-4967-8536-85ba89c9eec3\") "
Nov 03 18:06:28.168189 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:28.168168    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2458d282-5d5f-4967-8536-85ba89c9eec3-kube-api-access-xzrjh\" (UniqueName: \"kubernetes.io/projected/2458d282-5d5f-4967-8536-85ba89c9eec3-kube-api-access-xzrjh\") pod \"2458d282-5d5f-4967-8536-85ba89c9eec3\" (UID: \"2458d282-5d5f-4967-8536-85ba89c9eec3\") "
Nov 03 18:06:28.168283 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:28.168267    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2458d282-5d5f-4967-8536-85ba89c9eec3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2458d282-5d5f-4967-8536-85ba89c9eec3-maas-api-openshift-ca\") pod \"2458d282-5d5f-4967-8536-85ba89c9eec3\" (UID: \"2458d282-5d5f-4967-8536-85ba89c9eec3\") "
Nov 03 18:06:28.168955 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:28.168924    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2458d282-5d5f-4967-8536-85ba89c9eec3-maas-api-openshift-ca" pod "2458d282-5d5f-4967-8536-85ba89c9eec3" (UID: "2458d282-5d5f-4967-8536-85ba89c9eec3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:06:28.170819 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:28.170788    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2458d282-5d5f-4967-8536-85ba89c9eec3-maas-api-xks-ca" pod "2458d282-5d5f-4967-8536-85ba89c9eec3" (UID: "2458d282-5d5f-4967-8536-85ba89c9eec3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:06:28.171016 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:28.170990    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2458d282-5d5f-4967-8536-85ba89c9eec3-kube-api-access-xzrjh" pod "2458d282-5d5f-4967-8536-85ba89c9eec3" (UID: "2458d282-5d5f-4967-8536-85ba89c9eec3"). InnerVolumeSpecName "kube-api-access-xzrjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:06:28.281763 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:28.281712    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2458d282-5d5f-4967-8536-85ba89c9eec3-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:28.281763 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:28.281759    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2458d282-5d5f-4967-8536-85ba89c9eec3-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:28.281992 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:28.281775    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzrjh\" (UniqueName: \"kubernetes.io/projected/2458d282-5d5f-4967-8536-85ba89c9eec3-kube-api-access-xzrjh\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:28.954089 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:28.954054    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80bee53a-zmp7n"
Nov 03 18:06:28.954089 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:28.954054    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80bee53a-zmp7n" event={"ID":"2458d282-5d5f-4967-8536-85ba89c9eec3","Type":"ContainerDied","Data":"9c36a77e883a403611482be3f31a7f056d911246ceeb46a8582a3ad240d70549"}
Nov 03 18:06:28.954639 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:28.954148    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c36a77e883a403611482be3f31a7f056d911246ceeb46a8582a3ad240d70549"
Nov 03 18:06:29.066434 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:29.066095    5315 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 03 18:06:29.066434 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:29.066144    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4" podUID="c3b8cf40-02df-499d-b41c-6e4bd9283a3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 03 18:06:29.197449 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:29.197418    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4298e49-xjwqp"
Nov 03 18:06:29.294664 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:29.294628    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/518883f2-c175-4ad7-a4d0-15dfa5520315-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/518883f2-c175-4ad7-a4d0-15dfa5520315-maas-api-openshift-ca\") pod \"518883f2-c175-4ad7-a4d0-15dfa5520315\" (UID: \"518883f2-c175-4ad7-a4d0-15dfa5520315\") "
Nov 03 18:06:29.294865 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:29.294731    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/518883f2-c175-4ad7-a4d0-15dfa5520315-kube-api-access-vkg9h\" (UniqueName: \"kubernetes.io/projected/518883f2-c175-4ad7-a4d0-15dfa5520315-kube-api-access-vkg9h\") pod \"518883f2-c175-4ad7-a4d0-15dfa5520315\" (UID: \"518883f2-c175-4ad7-a4d0-15dfa5520315\") "
Nov 03 18:06:29.294982 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:29.294960    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/518883f2-c175-4ad7-a4d0-15dfa5520315-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/518883f2-c175-4ad7-a4d0-15dfa5520315-maas-api-xks-ca\") pod \"518883f2-c175-4ad7-a4d0-15dfa5520315\" (UID: \"518883f2-c175-4ad7-a4d0-15dfa5520315\") "
Nov 03 18:06:29.295055 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:29.295008    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/518883f2-c175-4ad7-a4d0-15dfa5520315-maas-api-openshift-ca" pod "518883f2-c175-4ad7-a4d0-15dfa5520315" (UID: "518883f2-c175-4ad7-a4d0-15dfa5520315"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:06:29.295492 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:29.295442    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/518883f2-c175-4ad7-a4d0-15dfa5520315-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:29.297377 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:29.297348    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/518883f2-c175-4ad7-a4d0-15dfa5520315-kube-api-access-vkg9h" pod "518883f2-c175-4ad7-a4d0-15dfa5520315" (UID: "518883f2-c175-4ad7-a4d0-15dfa5520315"). InnerVolumeSpecName "kube-api-access-vkg9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:06:29.297638 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:29.297586    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/518883f2-c175-4ad7-a4d0-15dfa5520315-maas-api-xks-ca" pod "518883f2-c175-4ad7-a4d0-15dfa5520315" (UID: "518883f2-c175-4ad7-a4d0-15dfa5520315"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:06:29.396841 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:29.396810    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkg9h\" (UniqueName: \"kubernetes.io/projected/518883f2-c175-4ad7-a4d0-15dfa5520315-kube-api-access-vkg9h\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:29.396841 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:29.396840    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/518883f2-c175-4ad7-a4d0-15dfa5520315-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:29.966840 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:29.966807    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4298e49-xjwqp"
Nov 03 18:06:29.967219 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:29.966952    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4298e49-xjwqp" event={"ID":"518883f2-c175-4ad7-a4d0-15dfa5520315","Type":"ContainerDied","Data":"a4d62659a7583e34e3d96dfa67e4bf4ae39508c9dc8b8c0a289c5ed03aa839ec"}
Nov 03 18:06:29.967219 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:29.966995    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4d62659a7583e34e3d96dfa67e4bf4ae39508c9dc8b8c0a289c5ed03aa839ec"
Nov 03 18:06:30.062224 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:30.062183    5315 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 03 18:06:30.062369 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:30.062247    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4" podUID="c3b8cf40-02df-499d-b41c-6e4bd9283a3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 03 18:06:30.976486 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:30.976447    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7353-c12ad0" event={"ID":"e34e974c-1009-4a89-9735-2493179d9411","Type":"ContainerStarted","Data":"fc494cb160de8ef74b8cf1c6e324d063a9ce199ac470e0a620e0d2a47c3cb4a7"}
Nov 03 18:06:31.062224 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:31.062183    5315 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 03 18:06:31.062408 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:31.062249    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4" podUID="c3b8cf40-02df-499d-b41c-6e4bd9283a3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 03 18:06:31.108313 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:31.108060    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7353-c12ad0" podStartSLOduration=2.115053265 podStartE2EDuration="6.10804625s" totalImagesPullingTime="3.992992985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:06:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:06:31.10192066 +0000 UTC m=+1230.118118730" watchObservedRunningTime="2026-11-03 18:06:31.10804625 +0000 UTC m=+1230.124244360"
Nov 03 18:06:32.062528 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:32.062491    5315 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 03 18:06:32.063013 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:32.062563    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4" podUID="c3b8cf40-02df-499d-b41c-6e4bd9283a3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 03 18:06:32.565489 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:32.565445    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7353-c12ad0"]
Nov 03 18:06:32.570423 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:32.570396    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7353-c12ad0"]
Nov 03 18:06:32.992371 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:32.992275    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7353-c12ad0" podUID="e34e974c-1009-4a89-9735-2493179d9411" containerName="test-curl-7353-c12ad0" containerID="cri-o://fc494cb160de8ef74b8cf1c6e324d063a9ce199ac470e0a620e0d2a47c3cb4a7" gracePeriod=2
Nov 03 18:06:33.049478 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:06:33.048917    5315 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7353-c12ad0\" is forbidden: User \"system:node:ip-10-0-128-204.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-204.us-west-2.compute.internal' and this object" podUID="e34e974c-1009-4a89-9735-2493179d9411" pod="opendatahub/test-curl-7353-c12ad0"
Nov 03 18:06:33.062019 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:33.061980    5315 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 03 18:06:33.062242 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:33.062211    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4" podUID="c3b8cf40-02df-499d-b41c-6e4bd9283a3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 03 18:06:34.027693 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:34.027656    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-683d0211-tgvbp"]
Nov 03 18:06:34.028282 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:34.028263    5315 cpu_manager.go:414] "containerMap: removing container" podUID="e34e974c-1009-4a89-9735-2493179d9411" containerName="test-curl-7353-c12ad0"
Nov 03 18:06:34.028368 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:34.028283    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e34e974c-1009-4a89-9735-2493179d9411" containerName="test-curl-7353-c12ad0"
Nov 03 18:06:34.028368 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:34.028303    5315 cpu_manager.go:414] "containerMap: removing container" podUID="518883f2-c175-4ad7-a4d0-15dfa5520315" containerName="revoke-keys"
Nov 03 18:06:34.028368 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:34.028311    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="518883f2-c175-4ad7-a4d0-15dfa5520315" containerName="revoke-keys"
Nov 03 18:06:34.028368 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:34.028362    5315 cpu_manager.go:414] "containerMap: removing container" podUID="2458d282-5d5f-4967-8536-85ba89c9eec3" containerName="revoke-keys"
Nov 03 18:06:34.028670 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:34.028371    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2458d282-5d5f-4967-8536-85ba89c9eec3" containerName="revoke-keys"
Nov 03 18:06:34.028670 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:34.028502    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="2458d282-5d5f-4967-8536-85ba89c9eec3" containerName="revoke-keys"
Nov 03 18:06:34.028670 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:34.028517    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="e34e974c-1009-4a89-9735-2493179d9411" containerName="test-curl-7353-c12ad0"
Nov 03 18:06:34.028670 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:34.028523    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="518883f2-c175-4ad7-a4d0-15dfa5520315" containerName="revoke-keys"
Nov 03 18:06:34.032672 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:34.032648    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-683d0211-tgvbp"
Nov 03 18:06:34.062140 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:34.062110    5315 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 03 18:06:34.062269 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:34.062183    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4" podUID="c3b8cf40-02df-499d-b41c-6e4bd9283a3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 03 18:06:34.081342 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:34.081308    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:06:34.081342 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:34.081308    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:06:34.127347 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:34.127313    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:06:34.127531 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:34.127313    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:06:34.131589 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:34.131552    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-683d0211-tgvbp"]
Nov 03 18:06:34.193302 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:34.193259    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e44f53a6-9661-4774-b376-61594363d002-maas-api-xks-ca\") pod \"maas-api-revoke-sub-683d0211-tgvbp\" (UID: \"e44f53a6-9661-4774-b376-61594363d002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-683d0211-tgvbp"
Nov 03 18:06:34.193578 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:34.193536    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e44f53a6-9661-4774-b376-61594363d002-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-683d0211-tgvbp\" (UID: \"e44f53a6-9661-4774-b376-61594363d002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-683d0211-tgvbp"
Nov 03 18:06:34.193856 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:34.193834    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67kzc\" (UniqueName: \"kubernetes.io/projected/e44f53a6-9661-4774-b376-61594363d002-kube-api-access-67kzc\") pod \"maas-api-revoke-sub-683d0211-tgvbp\" (UID: \"e44f53a6-9661-4774-b376-61594363d002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-683d0211-tgvbp"
Nov 03 18:06:34.295294 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:34.295191    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67kzc\" (UniqueName: \"kubernetes.io/projected/e44f53a6-9661-4774-b376-61594363d002-kube-api-access-67kzc\") pod \"maas-api-revoke-sub-683d0211-tgvbp\" (UID: \"e44f53a6-9661-4774-b376-61594363d002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-683d0211-tgvbp"
Nov 03 18:06:34.295294 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:34.295286    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e44f53a6-9661-4774-b376-61594363d002-maas-api-xks-ca\") pod \"maas-api-revoke-sub-683d0211-tgvbp\" (UID: \"e44f53a6-9661-4774-b376-61594363d002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-683d0211-tgvbp"
Nov 03 18:06:34.296595 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:34.295425    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e44f53a6-9661-4774-b376-61594363d002-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-683d0211-tgvbp\" (UID: \"e44f53a6-9661-4774-b376-61594363d002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-683d0211-tgvbp"
Nov 03 18:06:34.296595 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:34.296288    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e44f53a6-9661-4774-b376-61594363d002-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-683d0211-tgvbp\" (UID: \"e44f53a6-9661-4774-b376-61594363d002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-683d0211-tgvbp"
Nov 03 18:06:34.298480 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:34.298331    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e44f53a6-9661-4774-b376-61594363d002-maas-api-xks-ca\") pod \"maas-api-revoke-sub-683d0211-tgvbp\" (UID: \"e44f53a6-9661-4774-b376-61594363d002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-683d0211-tgvbp"
Nov 03 18:06:34.368631 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:34.362677    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67kzc\" (UniqueName: \"kubernetes.io/projected/e44f53a6-9661-4774-b376-61594363d002-kube-api-access-67kzc\") pod \"maas-api-revoke-sub-683d0211-tgvbp\" (UID: \"e44f53a6-9661-4774-b376-61594363d002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-683d0211-tgvbp"
Nov 03 18:06:34.646472 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:34.646380    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-683d0211-tgvbp"
Nov 03 18:06:35.032036 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:35.030920    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-683d0211-tgvbp"]
Nov 03 18:06:35.161828 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:35.161765    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4"
Nov 03 18:06:35.165200 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:35.162755    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4"
Nov 03 18:06:35.165200 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:35.164164    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4"
Nov 03 18:06:35.235103 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:35.235077    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7353-c12ad0"
Nov 03 18:06:35.315314 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:35.315286    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e34e974c-1009-4a89-9735-2493179d9411-kube-api-access-jqznk\" (UniqueName: \"kubernetes.io/projected/e34e974c-1009-4a89-9735-2493179d9411-kube-api-access-jqznk\") pod \"e34e974c-1009-4a89-9735-2493179d9411\" (UID: \"e34e974c-1009-4a89-9735-2493179d9411\") "
Nov 03 18:06:35.317878 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:35.317844    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e34e974c-1009-4a89-9735-2493179d9411-kube-api-access-jqznk" pod "e34e974c-1009-4a89-9735-2493179d9411" (UID: "e34e974c-1009-4a89-9735-2493179d9411"). InnerVolumeSpecName "kube-api-access-jqznk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:06:35.417328 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:35.417288    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqznk\" (UniqueName: \"kubernetes.io/projected/e34e974c-1009-4a89-9735-2493179d9411-kube-api-access-jqznk\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:36.021472 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:36.021436    5315 generic.go:356] "Generic (PLEG): container finished" podID="e34e974c-1009-4a89-9735-2493179d9411" containerID="fc494cb160de8ef74b8cf1c6e324d063a9ce199ac470e0a620e0d2a47c3cb4a7" exitCode=137
Nov 03 18:06:36.021655 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:36.021497    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7353-c12ad0"
Nov 03 18:06:36.021655 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:36.021496    5315 scope.go:122] "RemoveContainer" containerID="fc494cb160de8ef74b8cf1c6e324d063a9ce199ac470e0a620e0d2a47c3cb4a7"
Nov 03 18:06:36.023382 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:36.023356    5315 generic.go:356] "Generic (PLEG): container finished" podID="e44f53a6-9661-4774-b376-61594363d002" containerID="d04c61463bf9e0719fddcc2c45b1b15db890bd2c76e95be9d404456c268fc6dc" exitCode=0
Nov 03 18:06:36.023482 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:36.023458    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-683d0211-tgvbp" event={"ID":"e44f53a6-9661-4774-b376-61594363d002","Type":"ContainerDied","Data":"d04c61463bf9e0719fddcc2c45b1b15db890bd2c76e95be9d404456c268fc6dc"}
Nov 03 18:06:36.023528 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:36.023512    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-683d0211-tgvbp" event={"ID":"e44f53a6-9661-4774-b376-61594363d002","Type":"ContainerStarted","Data":"092fee78176b09769e44afd33b06a7b9204a3dcec1cedecb8b91081c8f5380d9"}
Nov 03 18:06:36.034079 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:36.034043    5315 scope.go:122] "RemoveContainer" containerID="fc494cb160de8ef74b8cf1c6e324d063a9ce199ac470e0a620e0d2a47c3cb4a7"
Nov 03 18:06:36.038419 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:06:36.037078    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc494cb160de8ef74b8cf1c6e324d063a9ce199ac470e0a620e0d2a47c3cb4a7\": container with ID starting with fc494cb160de8ef74b8cf1c6e324d063a9ce199ac470e0a620e0d2a47c3cb4a7 not found: ID does not exist" containerID="fc494cb160de8ef74b8cf1c6e324d063a9ce199ac470e0a620e0d2a47c3cb4a7"
Nov 03 18:06:36.038419 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:36.037129    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc494cb160de8ef74b8cf1c6e324d063a9ce199ac470e0a620e0d2a47c3cb4a7"} err="failed to get container status \"fc494cb160de8ef74b8cf1c6e324d063a9ce199ac470e0a620e0d2a47c3cb4a7\": rpc error: code = NotFound desc = could not find container \"fc494cb160de8ef74b8cf1c6e324d063a9ce199ac470e0a620e0d2a47c3cb4a7\": container with ID starting with fc494cb160de8ef74b8cf1c6e324d063a9ce199ac470e0a620e0d2a47c3cb4a7 not found: ID does not exist"
Nov 03 18:06:36.060461 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:36.060427    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bea68c5-m9c4r"]
Nov 03 18:06:36.090060 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:36.090025    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bea68c5-m9c4r"]
Nov 03 18:06:36.208583 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:06:36.208490    5315 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7353-c12ad0\" is forbidden: User \"system:node:ip-10-0-128-204.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-204.us-west-2.compute.internal' and this object" podUID="e34e974c-1009-4a89-9735-2493179d9411" pod="opendatahub/test-curl-7353-c12ad0"
Nov 03 18:06:36.527823 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:36.527782    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns"]
Nov 03 18:06:36.532706 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:36.532673    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns"
Nov 03 18:06:36.737865 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:36.737799    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-workload-certs\") pod \"e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns\" (UID: \"8832b3d8-1880-4277-8a0c-f9bf94eef7b3\") " pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns"
Nov 03 18:06:36.738416 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:36.738373    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-istio-envoy\") pod \"e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns\" (UID: \"8832b3d8-1880-4277-8a0c-f9bf94eef7b3\") " pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns"
Nov 03 18:06:36.745631 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:36.738805    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-86101f5d-gw-openshift-default-dockercfg-dh67h\""
Nov 03 18:06:36.745631 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:36.745429    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-istio-token\") pod \"e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns\" (UID: \"8832b3d8-1880-4277-8a0c-f9bf94eef7b3\") " pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns"
Nov 03 18:06:36.745861 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:36.745718    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-workload-socket\") pod \"e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns\" (UID: \"8832b3d8-1880-4277-8a0c-f9bf94eef7b3\") " pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns"
Nov 03 18:06:36.745861 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:36.745793    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4zxz\" (UniqueName: \"kubernetes.io/projected/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-kube-api-access-n4zxz\") pod \"e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns\" (UID: \"8832b3d8-1880-4277-8a0c-f9bf94eef7b3\") " pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns"
Nov 03 18:06:36.745975 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:36.745872    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-istio-podinfo\") pod \"e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns\" (UID: \"8832b3d8-1880-4277-8a0c-f9bf94eef7b3\") " pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns"
Nov 03 18:06:36.746041 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:36.746026    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-istiod-ca-cert\") pod \"e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns\" (UID: \"8832b3d8-1880-4277-8a0c-f9bf94eef7b3\") " pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns"
Nov 03 18:06:36.755076 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:36.746101    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-credential-socket\") pod \"e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns\" (UID: \"8832b3d8-1880-4277-8a0c-f9bf94eef7b3\") " pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns"
Nov 03 18:06:36.755076 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:36.746195    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-istio-data\") pod \"e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns\" (UID: \"8832b3d8-1880-4277-8a0c-f9bf94eef7b3\") " pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns"
Nov 03 18:06:36.787689 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:06:36.786439    5315 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7353-c12ad0\" is forbidden: User \"system:node:ip-10-0-128-204.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-204.us-west-2.compute.internal' and this object" podUID="e34e974c-1009-4a89-9735-2493179d9411" pod="opendatahub/test-curl-7353-c12ad0"
Nov 03 18:06:36.825598 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:36.825561    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns"]
Nov 03 18:06:36.953282 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:36.953245    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-workload-socket\") pod \"e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns\" (UID: \"8832b3d8-1880-4277-8a0c-f9bf94eef7b3\") " pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns"
Nov 03 18:06:36.955633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:36.953557    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4zxz\" (UniqueName: \"kubernetes.io/projected/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-kube-api-access-n4zxz\") pod \"e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns\" (UID: \"8832b3d8-1880-4277-8a0c-f9bf94eef7b3\") " pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns"
Nov 03 18:06:36.955923 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:36.955895    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-istio-podinfo\") pod \"e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns\" (UID: \"8832b3d8-1880-4277-8a0c-f9bf94eef7b3\") " pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns"
Nov 03 18:06:36.956152 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:36.956133    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-istiod-ca-cert\") pod \"e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns\" (UID: \"8832b3d8-1880-4277-8a0c-f9bf94eef7b3\") " pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns"
Nov 03 18:06:36.956306 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:36.956283    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-credential-socket\") pod \"e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns\" (UID: \"8832b3d8-1880-4277-8a0c-f9bf94eef7b3\") " pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns"
Nov 03 18:06:36.956405 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:36.956366    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-istio-data\") pod \"e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns\" (UID: \"8832b3d8-1880-4277-8a0c-f9bf94eef7b3\") " pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns"
Nov 03 18:06:36.956881 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:36.956477    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-workload-certs\") pod \"e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns\" (UID: \"8832b3d8-1880-4277-8a0c-f9bf94eef7b3\") " pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns"
Nov 03 18:06:36.956881 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:36.956509    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-istio-envoy\") pod \"e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns\" (UID: \"8832b3d8-1880-4277-8a0c-f9bf94eef7b3\") " pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns"
Nov 03 18:06:36.956881 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:36.956564    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-istio-token\") pod \"e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns\" (UID: \"8832b3d8-1880-4277-8a0c-f9bf94eef7b3\") " pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns"
Nov 03 18:06:36.957424 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:36.957372    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-workload-socket\") pod \"e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns\" (UID: \"8832b3d8-1880-4277-8a0c-f9bf94eef7b3\") " pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns"
Nov 03 18:06:36.958977 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:36.958950    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-credential-socket\") pod \"e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns\" (UID: \"8832b3d8-1880-4277-8a0c-f9bf94eef7b3\") " pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns"
Nov 03 18:06:36.959340 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:36.959318    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-workload-certs\") pod \"e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns\" (UID: \"8832b3d8-1880-4277-8a0c-f9bf94eef7b3\") " pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns"
Nov 03 18:06:36.959520 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:36.959504    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-istio-data\") pod \"e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns\" (UID: \"8832b3d8-1880-4277-8a0c-f9bf94eef7b3\") " pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns"
Nov 03 18:06:36.959852 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:36.959698    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-istiod-ca-cert\") pod \"e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns\" (UID: \"8832b3d8-1880-4277-8a0c-f9bf94eef7b3\") " pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns"
Nov 03 18:06:37.001670 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:36.995483    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-istio-podinfo\") pod \"e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns\" (UID: \"8832b3d8-1880-4277-8a0c-f9bf94eef7b3\") " pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns"
Nov 03 18:06:37.001670 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:36.996859    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-istio-envoy\") pod \"e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns\" (UID: \"8832b3d8-1880-4277-8a0c-f9bf94eef7b3\") " pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns"
Nov 03 18:06:37.055381 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:37.055275    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-istio-token\") pod \"e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns\" (UID: \"8832b3d8-1880-4277-8a0c-f9bf94eef7b3\") " pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns"
Nov 03 18:06:37.056113 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:37.055953    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4zxz\" (UniqueName: \"kubernetes.io/projected/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-kube-api-access-n4zxz\") pod \"e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns\" (UID: \"8832b3d8-1880-4277-8a0c-f9bf94eef7b3\") " pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns"
Nov 03 18:06:37.117383 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:37.117346    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85c0fe57-c156-4fb6-a72f-e3faabada5d9" path="/var/lib/kubelet/pods/85c0fe57-c156-4fb6-a72f-e3faabada5d9/volumes"
Nov 03 18:06:37.117939 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:37.117913    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e34e974c-1009-4a89-9735-2493179d9411" path="/var/lib/kubelet/pods/e34e974c-1009-4a89-9735-2493179d9411/volumes"
Nov 03 18:06:37.147896 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:37.147863    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns"
Nov 03 18:06:37.255444 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:37.255412    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-683d0211-tgvbp"
Nov 03 18:06:37.365322 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:37.365235    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e44f53a6-9661-4774-b376-61594363d002-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e44f53a6-9661-4774-b376-61594363d002-maas-api-xks-ca\") pod \"e44f53a6-9661-4774-b376-61594363d002\" (UID: \"e44f53a6-9661-4774-b376-61594363d002\") "
Nov 03 18:06:37.365487 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:37.365382    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e44f53a6-9661-4774-b376-61594363d002-kube-api-access-67kzc\" (UniqueName: \"kubernetes.io/projected/e44f53a6-9661-4774-b376-61594363d002-kube-api-access-67kzc\") pod \"e44f53a6-9661-4774-b376-61594363d002\" (UID: \"e44f53a6-9661-4774-b376-61594363d002\") "
Nov 03 18:06:37.365487 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:37.365454    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e44f53a6-9661-4774-b376-61594363d002-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e44f53a6-9661-4774-b376-61594363d002-maas-api-openshift-ca\") pod \"e44f53a6-9661-4774-b376-61594363d002\" (UID: \"e44f53a6-9661-4774-b376-61594363d002\") "
Nov 03 18:06:37.365816 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:37.365790    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e44f53a6-9661-4774-b376-61594363d002-maas-api-openshift-ca" pod "e44f53a6-9661-4774-b376-61594363d002" (UID: "e44f53a6-9661-4774-b376-61594363d002"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:06:37.366157 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:37.366122    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e44f53a6-9661-4774-b376-61594363d002-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:37.367518 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:37.367492    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e44f53a6-9661-4774-b376-61594363d002-maas-api-xks-ca" pod "e44f53a6-9661-4774-b376-61594363d002" (UID: "e44f53a6-9661-4774-b376-61594363d002"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:06:37.367883 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:37.367855    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e44f53a6-9661-4774-b376-61594363d002-kube-api-access-67kzc" pod "e44f53a6-9661-4774-b376-61594363d002" (UID: "e44f53a6-9661-4774-b376-61594363d002"). InnerVolumeSpecName "kube-api-access-67kzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:06:37.467257 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:37.467220    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67kzc\" (UniqueName: \"kubernetes.io/projected/e44f53a6-9661-4774-b376-61594363d002-kube-api-access-67kzc\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:37.467257 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:37.467251    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e44f53a6-9661-4774-b376-61594363d002-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:37.605228 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:37.605197    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns"]
Nov 03 18:06:37.609515 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:06:37.609483    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8832b3d8_1880_4277_8a0c_f9bf94eef7b3.slice/crio-42072296ac0af62669dda6e644897cc9961e262f2a0c97c796549471401d2c5e WatchSource:0}: Error finding container 42072296ac0af62669dda6e644897cc9961e262f2a0c97c796549471401d2c5e: Status 404 returned error can't find the container with id 42072296ac0af62669dda6e644897cc9961e262f2a0c97c796549471401d2c5e
Nov 03 18:06:37.612099 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:37.612067    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:06:37.612159 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:37.612136    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:06:37.612192 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:37.612168    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:06:38.074753 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:38.074711    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns" event={"ID":"8832b3d8-1880-4277-8a0c-f9bf94eef7b3","Type":"ContainerStarted","Data":"767d0ab15d11016a2d5140fad23874f7304cd7ecc9136f36983a08713a478858"}
Nov 03 18:06:38.075153 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:38.074785    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns" event={"ID":"8832b3d8-1880-4277-8a0c-f9bf94eef7b3","Type":"ContainerStarted","Data":"42072296ac0af62669dda6e644897cc9961e262f2a0c97c796549471401d2c5e"}
Nov 03 18:06:38.076808 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:38.076781    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-683d0211-tgvbp"
Nov 03 18:06:38.076926 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:38.076806    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-683d0211-tgvbp" event={"ID":"e44f53a6-9661-4774-b376-61594363d002","Type":"ContainerDied","Data":"092fee78176b09769e44afd33b06a7b9204a3dcec1cedecb8b91081c8f5380d9"}
Nov 03 18:06:38.076926 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:38.076849    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="092fee78176b09769e44afd33b06a7b9204a3dcec1cedecb8b91081c8f5380d9"
Nov 03 18:06:38.148629 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:38.148538    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns"
Nov 03 18:06:38.155777 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:38.155745    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns"
Nov 03 18:06:38.188745 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:38.188696    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns" podStartSLOduration=2.188676925 podStartE2EDuration="2.188676925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:06:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:06:38.188214594 +0000 UTC m=+1237.204412665" watchObservedRunningTime="2026-11-03 18:06:38.188676925 +0000 UTC m=+1237.204874988"
Nov 03 18:06:39.060814 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:39.060777    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30d44741-mvwv6"]
Nov 03 18:06:39.070325 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:39.070296    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30d44741-mvwv6"]
Nov 03 18:06:39.086547 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:39.086516    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns"
Nov 03 18:06:39.087589 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:39.087563    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns"
Nov 03 18:06:39.104871 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:39.104841    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8eea994-0b05-4bad-8342-b9652610fe74" path="/var/lib/kubelet/pods/a8eea994-0b05-4bad-8342-b9652610fe74/volumes"
Nov 03 18:06:40.113133 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:40.113097    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca9298d0-wkhpc"]
Nov 03 18:06:40.114261 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:40.113986    5315 cpu_manager.go:414] "containerMap: removing container" podUID="e44f53a6-9661-4774-b376-61594363d002" containerName="revoke-keys"
Nov 03 18:06:40.114261 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:40.114017    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e44f53a6-9661-4774-b376-61594363d002" containerName="revoke-keys"
Nov 03 18:06:40.114261 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:40.114222    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="e44f53a6-9661-4774-b376-61594363d002" containerName="revoke-keys"
Nov 03 18:06:40.119173 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:40.119146    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca9298d0-wkhpc"
Nov 03 18:06:40.181477 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:40.181419    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:06:40.181694 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:40.181571    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:06:40.236018 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:40.235985    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca9298d0-wkhpc"]
Nov 03 18:06:40.236219 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:40.236144    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:06:40.236219 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:40.236187    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:06:40.286028 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:40.285988    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80bf3ec8-d27c-4ddd-a862-f5c405b11349-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca9298d0-wkhpc\" (UID: \"80bf3ec8-d27c-4ddd-a862-f5c405b11349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca9298d0-wkhpc"
Nov 03 18:06:40.286225 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:40.286069    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5vjz\" (UniqueName: \"kubernetes.io/projected/80bf3ec8-d27c-4ddd-a862-f5c405b11349-kube-api-access-d5vjz\") pod \"maas-api-revoke-sub-ca9298d0-wkhpc\" (UID: \"80bf3ec8-d27c-4ddd-a862-f5c405b11349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca9298d0-wkhpc"
Nov 03 18:06:40.286393 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:40.286372    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80bf3ec8-d27c-4ddd-a862-f5c405b11349-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca9298d0-wkhpc\" (UID: \"80bf3ec8-d27c-4ddd-a862-f5c405b11349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca9298d0-wkhpc"
Nov 03 18:06:40.387738 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:40.387643    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80bf3ec8-d27c-4ddd-a862-f5c405b11349-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca9298d0-wkhpc\" (UID: \"80bf3ec8-d27c-4ddd-a862-f5c405b11349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca9298d0-wkhpc"
Nov 03 18:06:40.387914 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:40.387750    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80bf3ec8-d27c-4ddd-a862-f5c405b11349-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca9298d0-wkhpc\" (UID: \"80bf3ec8-d27c-4ddd-a862-f5c405b11349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca9298d0-wkhpc"
Nov 03 18:06:40.387914 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:40.387802    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5vjz\" (UniqueName: \"kubernetes.io/projected/80bf3ec8-d27c-4ddd-a862-f5c405b11349-kube-api-access-d5vjz\") pod \"maas-api-revoke-sub-ca9298d0-wkhpc\" (UID: \"80bf3ec8-d27c-4ddd-a862-f5c405b11349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca9298d0-wkhpc"
Nov 03 18:06:40.388641 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:40.388586    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80bf3ec8-d27c-4ddd-a862-f5c405b11349-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca9298d0-wkhpc\" (UID: \"80bf3ec8-d27c-4ddd-a862-f5c405b11349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca9298d0-wkhpc"
Nov 03 18:06:40.390269 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:40.390244    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80bf3ec8-d27c-4ddd-a862-f5c405b11349-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca9298d0-wkhpc\" (UID: \"80bf3ec8-d27c-4ddd-a862-f5c405b11349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca9298d0-wkhpc"
Nov 03 18:06:40.447059 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:40.447023    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5vjz\" (UniqueName: \"kubernetes.io/projected/80bf3ec8-d27c-4ddd-a862-f5c405b11349-kube-api-access-d5vjz\") pod \"maas-api-revoke-sub-ca9298d0-wkhpc\" (UID: \"80bf3ec8-d27c-4ddd-a862-f5c405b11349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca9298d0-wkhpc"
Nov 03 18:06:40.449873 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:40.449847    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca9298d0-wkhpc"
Nov 03 18:06:40.829074 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:40.829044    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca9298d0-wkhpc"]
Nov 03 18:06:40.833677 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:06:40.833645    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80bf3ec8_d27c_4ddd_a862_f5c405b11349.slice/crio-ef759037aa597293ccf2b7c76332983d0725abbf82dcc94d66af4e8d3f3507d6 WatchSource:0}: Error finding container ef759037aa597293ccf2b7c76332983d0725abbf82dcc94d66af4e8d3f3507d6: Status 404 returned error can't find the container with id ef759037aa597293ccf2b7c76332983d0725abbf82dcc94d66af4e8d3f3507d6
Nov 03 18:06:41.106376 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:41.106300    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca9298d0-wkhpc" event={"ID":"80bf3ec8-d27c-4ddd-a862-f5c405b11349","Type":"ContainerStarted","Data":"ef759037aa597293ccf2b7c76332983d0725abbf82dcc94d66af4e8d3f3507d6"}
Nov 03 18:06:42.114408 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:42.114321    5315 generic.go:356] "Generic (PLEG): container finished" podID="80bf3ec8-d27c-4ddd-a862-f5c405b11349" containerID="314ef8a11b70c4fb76504000b9c052bb100bd7709a84fb89d6b905e591643b91" exitCode=0
Nov 03 18:06:42.114810 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:42.114409    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca9298d0-wkhpc" event={"ID":"80bf3ec8-d27c-4ddd-a862-f5c405b11349","Type":"ContainerDied","Data":"314ef8a11b70c4fb76504000b9c052bb100bd7709a84fb89d6b905e591643b91"}
Nov 03 18:06:43.272055 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:43.272027    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca9298d0-wkhpc"
Nov 03 18:06:43.345291 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:43.345251    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80bf3ec8-d27c-4ddd-a862-f5c405b11349-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80bf3ec8-d27c-4ddd-a862-f5c405b11349-maas-api-openshift-ca\") pod \"80bf3ec8-d27c-4ddd-a862-f5c405b11349\" (UID: \"80bf3ec8-d27c-4ddd-a862-f5c405b11349\") "
Nov 03 18:06:43.345922 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:43.345309    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80bf3ec8-d27c-4ddd-a862-f5c405b11349-kube-api-access-d5vjz\" (UniqueName: \"kubernetes.io/projected/80bf3ec8-d27c-4ddd-a862-f5c405b11349-kube-api-access-d5vjz\") pod \"80bf3ec8-d27c-4ddd-a862-f5c405b11349\" (UID: \"80bf3ec8-d27c-4ddd-a862-f5c405b11349\") "
Nov 03 18:06:43.345922 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:43.345546    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80bf3ec8-d27c-4ddd-a862-f5c405b11349-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80bf3ec8-d27c-4ddd-a862-f5c405b11349-maas-api-xks-ca\") pod \"80bf3ec8-d27c-4ddd-a862-f5c405b11349\" (UID: \"80bf3ec8-d27c-4ddd-a862-f5c405b11349\") "
Nov 03 18:06:43.346832 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:43.346483    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80bf3ec8-d27c-4ddd-a862-f5c405b11349-maas-api-openshift-ca" pod "80bf3ec8-d27c-4ddd-a862-f5c405b11349" (UID: "80bf3ec8-d27c-4ddd-a862-f5c405b11349"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:06:43.346832 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:43.346657    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80bf3ec8-d27c-4ddd-a862-f5c405b11349-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:43.349224 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:43.349172    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80bf3ec8-d27c-4ddd-a862-f5c405b11349-kube-api-access-d5vjz" pod "80bf3ec8-d27c-4ddd-a862-f5c405b11349" (UID: "80bf3ec8-d27c-4ddd-a862-f5c405b11349"). InnerVolumeSpecName "kube-api-access-d5vjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:06:43.349416 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:43.349395    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80bf3ec8-d27c-4ddd-a862-f5c405b11349-maas-api-xks-ca" pod "80bf3ec8-d27c-4ddd-a862-f5c405b11349" (UID: "80bf3ec8-d27c-4ddd-a862-f5c405b11349"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:06:43.448221 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:43.448134    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5vjz\" (UniqueName: \"kubernetes.io/projected/80bf3ec8-d27c-4ddd-a862-f5c405b11349-kube-api-access-d5vjz\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:43.448221 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:43.448165    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80bf3ec8-d27c-4ddd-a862-f5c405b11349-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:44.132207 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:44.132171    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca9298d0-wkhpc" event={"ID":"80bf3ec8-d27c-4ddd-a862-f5c405b11349","Type":"ContainerDied","Data":"ef759037aa597293ccf2b7c76332983d0725abbf82dcc94d66af4e8d3f3507d6"}
Nov 03 18:06:44.132375 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:44.132226    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef759037aa597293ccf2b7c76332983d0725abbf82dcc94d66af4e8d3f3507d6"
Nov 03 18:06:44.132375 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:44.132184    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca9298d0-wkhpc"
Nov 03 18:06:44.250833 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:44.250792    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaaeec56-qxbj2"]
Nov 03 18:06:44.251462 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:44.251440    5315 cpu_manager.go:414] "containerMap: removing container" podUID="80bf3ec8-d27c-4ddd-a862-f5c405b11349" containerName="revoke-keys"
Nov 03 18:06:44.251561 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:44.251463    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80bf3ec8-d27c-4ddd-a862-f5c405b11349" containerName="revoke-keys"
Nov 03 18:06:44.251652 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:44.251636    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="80bf3ec8-d27c-4ddd-a862-f5c405b11349" containerName="revoke-keys"
Nov 03 18:06:44.255300 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:44.255279    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaaeec56-qxbj2"
Nov 03 18:06:44.304789 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:44.304760    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:06:44.304789 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:44.304764    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:06:44.305195 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:44.304791    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:06:44.305195 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:44.304818    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:06:44.307077 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:44.307047    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaaeec56-qxbj2"]
Nov 03 18:06:44.367003 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:44.366968    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fftm\" (UniqueName: \"kubernetes.io/projected/1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef-kube-api-access-9fftm\") pod \"maas-api-revoke-sub-aaaeec56-qxbj2\" (UID: \"1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaaeec56-qxbj2"
Nov 03 18:06:44.367183 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:44.367011    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaaeec56-qxbj2\" (UID: \"1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaaeec56-qxbj2"
Nov 03 18:06:44.367303 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:44.367236    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaaeec56-qxbj2\" (UID: \"1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaaeec56-qxbj2"
Nov 03 18:06:44.468377 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:44.468286    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fftm\" (UniqueName: \"kubernetes.io/projected/1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef-kube-api-access-9fftm\") pod \"maas-api-revoke-sub-aaaeec56-qxbj2\" (UID: \"1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaaeec56-qxbj2"
Nov 03 18:06:44.468377 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:44.468328    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaaeec56-qxbj2\" (UID: \"1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaaeec56-qxbj2"
Nov 03 18:06:44.468675 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:44.468422    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaaeec56-qxbj2\" (UID: \"1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaaeec56-qxbj2"
Nov 03 18:06:44.469177 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:44.469151    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaaeec56-qxbj2\" (UID: \"1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaaeec56-qxbj2"
Nov 03 18:06:44.470837 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:44.470814    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaaeec56-qxbj2\" (UID: \"1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaaeec56-qxbj2"
Nov 03 18:06:44.525523 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:44.525488    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fftm\" (UniqueName: \"kubernetes.io/projected/1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef-kube-api-access-9fftm\") pod \"maas-api-revoke-sub-aaaeec56-qxbj2\" (UID: \"1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaaeec56-qxbj2"
Nov 03 18:06:44.568959 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:44.568919    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaaeec56-qxbj2"
Nov 03 18:06:44.852335 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:44.852296    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c9ad588-jp7t8"]
Nov 03 18:06:44.857845 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:44.857823    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9ad588-jp7t8"
Nov 03 18:06:44.924874 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:44.924829    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c9ad588-jp7t8"]
Nov 03 18:06:44.971152 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:44.969033    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaaeec56-qxbj2"]
Nov 03 18:06:44.971152 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:06:44.970355    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fb39614_be67_47c2_a0a6_dd8d1bd6b2ef.slice/crio-950d6e871455ef6bec279e0b8a24bfee7e52cc038360213b65b3c8add5467844 WatchSource:0}: Error finding container 950d6e871455ef6bec279e0b8a24bfee7e52cc038360213b65b3c8add5467844: Status 404 returned error can't find the container with id 950d6e871455ef6bec279e0b8a24bfee7e52cc038360213b65b3c8add5467844
Nov 03 18:06:44.973779 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:44.973746    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/051dcdb7-a6c4-4167-b814-8b16bb04fb4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c9ad588-jp7t8\" (UID: \"051dcdb7-a6c4-4167-b814-8b16bb04fb4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9ad588-jp7t8"
Nov 03 18:06:44.973921 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:44.973898    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/051dcdb7-a6c4-4167-b814-8b16bb04fb4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c9ad588-jp7t8\" (UID: \"051dcdb7-a6c4-4167-b814-8b16bb04fb4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9ad588-jp7t8"
Nov 03 18:06:44.974453 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:44.974420    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g72ft\" (UniqueName: \"kubernetes.io/projected/051dcdb7-a6c4-4167-b814-8b16bb04fb4e-kube-api-access-g72ft\") pod \"maas-api-revoke-sub-1c9ad588-jp7t8\" (UID: \"051dcdb7-a6c4-4167-b814-8b16bb04fb4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9ad588-jp7t8"
Nov 03 18:06:45.075932 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:45.075890    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/051dcdb7-a6c4-4167-b814-8b16bb04fb4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c9ad588-jp7t8\" (UID: \"051dcdb7-a6c4-4167-b814-8b16bb04fb4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9ad588-jp7t8"
Nov 03 18:06:45.076129 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:45.076015    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/051dcdb7-a6c4-4167-b814-8b16bb04fb4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c9ad588-jp7t8\" (UID: \"051dcdb7-a6c4-4167-b814-8b16bb04fb4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9ad588-jp7t8"
Nov 03 18:06:45.076129 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:45.076090    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g72ft\" (UniqueName: \"kubernetes.io/projected/051dcdb7-a6c4-4167-b814-8b16bb04fb4e-kube-api-access-g72ft\") pod \"maas-api-revoke-sub-1c9ad588-jp7t8\" (UID: \"051dcdb7-a6c4-4167-b814-8b16bb04fb4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9ad588-jp7t8"
Nov 03 18:06:45.076745 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:45.076719    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/051dcdb7-a6c4-4167-b814-8b16bb04fb4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c9ad588-jp7t8\" (UID: \"051dcdb7-a6c4-4167-b814-8b16bb04fb4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9ad588-jp7t8"
Nov 03 18:06:45.078379 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:45.078356    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/051dcdb7-a6c4-4167-b814-8b16bb04fb4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c9ad588-jp7t8\" (UID: \"051dcdb7-a6c4-4167-b814-8b16bb04fb4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9ad588-jp7t8"
Nov 03 18:06:45.136222 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:45.136151    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g72ft\" (UniqueName: \"kubernetes.io/projected/051dcdb7-a6c4-4167-b814-8b16bb04fb4e-kube-api-access-g72ft\") pod \"maas-api-revoke-sub-1c9ad588-jp7t8\" (UID: \"051dcdb7-a6c4-4167-b814-8b16bb04fb4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9ad588-jp7t8"
Nov 03 18:06:45.141068 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:45.141039    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaaeec56-qxbj2" event={"ID":"1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef","Type":"ContainerStarted","Data":"950d6e871455ef6bec279e0b8a24bfee7e52cc038360213b65b3c8add5467844"}
Nov 03 18:06:45.172726 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:45.172698    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9ad588-jp7t8"
Nov 03 18:06:45.551355 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:45.551312    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c9ad588-jp7t8"]
Nov 03 18:06:45.555801 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:06:45.555772    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod051dcdb7_a6c4_4167_b814_8b16bb04fb4e.slice/crio-dafdd6fdb06cc31a3d278a3634a815ca1e29e0b87907998fd35b2eb598bfd113 WatchSource:0}: Error finding container dafdd6fdb06cc31a3d278a3634a815ca1e29e0b87907998fd35b2eb598bfd113: Status 404 returned error can't find the container with id dafdd6fdb06cc31a3d278a3634a815ca1e29e0b87907998fd35b2eb598bfd113
Nov 03 18:06:46.201688 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:46.186758    5315 generic.go:356] "Generic (PLEG): container finished" podID="1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef" containerID="becc088e90162c524ca741adf4135fcbe7db9021e2dadb79e3ff33aa8f9d20f0" exitCode=0
Nov 03 18:06:46.201688 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:46.186899    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaaeec56-qxbj2" event={"ID":"1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef","Type":"ContainerDied","Data":"becc088e90162c524ca741adf4135fcbe7db9021e2dadb79e3ff33aa8f9d20f0"}
Nov 03 18:06:46.212631 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:46.206815    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9ad588-jp7t8" event={"ID":"051dcdb7-a6c4-4167-b814-8b16bb04fb4e","Type":"ContainerStarted","Data":"af04986322ab502bfde031edaab5efacd4b5f367d4223c7dabaa06654be3c6ba"}
Nov 03 18:06:46.212631 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:46.206898    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9ad588-jp7t8" event={"ID":"051dcdb7-a6c4-4167-b814-8b16bb04fb4e","Type":"ContainerStarted","Data":"dafdd6fdb06cc31a3d278a3634a815ca1e29e0b87907998fd35b2eb598bfd113"}
Nov 03 18:06:46.452693 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:46.450737    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9ad588-jp7t8" podStartSLOduration=2.450710776 podStartE2EDuration="2.450710776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:06:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:06:46.448157273 +0000 UTC m=+1245.464355343" watchObservedRunningTime="2026-11-03 18:06:46.450710776 +0000 UTC m=+1245.466908908"
Nov 03 18:06:47.275689 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:47.272797    5315 generic.go:356] "Generic (PLEG): container finished" podID="051dcdb7-a6c4-4167-b814-8b16bb04fb4e" containerID="af04986322ab502bfde031edaab5efacd4b5f367d4223c7dabaa06654be3c6ba" exitCode=0
Nov 03 18:06:47.275689 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:47.273454    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9ad588-jp7t8" event={"ID":"051dcdb7-a6c4-4167-b814-8b16bb04fb4e","Type":"ContainerDied","Data":"af04986322ab502bfde031edaab5efacd4b5f367d4223c7dabaa06654be3c6ba"}
Nov 03 18:06:47.339643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:47.338205    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c" podUID="8ac59539-8ac1-466e-8d1b-b2be4c42c708" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:47.380639 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:47.375711    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7" podUID="33725a51-6459-4f15-a866-e9dee41a9732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:47.489634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:47.488718    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv" podUID="f6d706cb-3aa5-419c-8f6c-29347bb788b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:47.560942 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:47.557580    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7" podUID="69760dba-faf9-4c08-895c-ad70fe95fd49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:47.661693 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:47.649713    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l" podUID="114dcf54-e4eb-4294-a77b-604fe6d57d48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:47.770636 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:47.768145    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr" podUID="df8e0b71-e876-4cea-a7ae-3532c887c660" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:47.770636 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:47.768953    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7" podUID="09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:48.033959 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:48.033894    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr" podUID="08159a02-fc2e-4138-824d-be5ba3e141d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:48.114740 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:48.114686    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt" podUID="7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:48.214870 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:48.167051    5315 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv" podUID="f6d706cb-3aa5-419c-8f6c-29347bb788b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": context deadline exceeded"
Nov 03 18:06:48.387943 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:48.387888    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7" podUID="33725a51-6459-4f15-a866-e9dee41a9732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:48.388635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:48.388597    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7"
Nov 03 18:06:48.390496 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:48.390466    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c" podUID="8ac59539-8ac1-466e-8d1b-b2be4c42c708" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Nov 03 18:06:48.391277 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:48.391256    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c"
Nov 03 18:06:48.498508 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:48.496996    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv" podUID="f6d706cb-3aa5-419c-8f6c-29347bb788b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:48.498508 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:48.497909    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv"
Nov 03 18:06:48.575676 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:48.573841    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaaeec56-qxbj2"
Nov 03 18:06:48.575676 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:48.574000    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7" podUID="69760dba-faf9-4c08-895c-ad70fe95fd49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:48.580835 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:48.580789    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7"
Nov 03 18:06:48.659231 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:48.659179    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l" podUID="114dcf54-e4eb-4294-a77b-604fe6d57d48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:48.753173 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:48.682083    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s" podUID="155e4b39-1877-4bd9-8541-5d6bd3aa0a8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:48.771115 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:48.754435    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l"
Nov 03 18:06:48.771872 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:48.771839    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7" podUID="09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Nov 03 18:06:49.214728 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:48.772140    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr" podUID="df8e0b71-e876-4cea-a7ae-3532c887c660" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 18:06:49.214728 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:49.154660    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef-maas-api-openshift-ca\") pod \"1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef\" (UID: \"1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef\") "
Nov 03 18:06:49.214728 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:49.204503    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef-kube-api-access-9fftm\" (UniqueName: \"kubernetes.io/projected/1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef-kube-api-access-9fftm\") pod \"1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef\" (UID: \"1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef\") "
Nov 03 18:06:49.214728 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:49.204713    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef-maas-api-xks-ca\") pod \"1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef\" (UID: \"1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef\") "
Nov 03 18:06:49.236677 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:49.224906    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q" podUID="c974b0f1-e00b-4e4a-b193-df769068983d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 18:06:49.303171 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:49.237639    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef-maas-api-openshift-ca" pod "1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef" (UID: "1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:06:49.306695 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:49.188721    5315 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7" podUID="09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:49.410947 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:49.406595    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c" podUID="8ac59539-8ac1-466e-8d1b-b2be4c42c708" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Nov 03 18:06:49.519667 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:49.494635    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7" podUID="33725a51-6459-4f15-a866-e9dee41a9732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 03 18:06:49.531654 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:49.529808    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv" podUID="f6d706cb-3aa5-419c-8f6c-29347bb788b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 18:06:49.531654 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:49.531628    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr" podUID="08159a02-fc2e-4138-824d-be5ba3e141d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:49.561924 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:49.203976    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt" podUID="7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Nov 03 18:06:49.820688 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:49.803976    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef-maas-api-xks-ca" pod "1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef" (UID: "1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:06:50.191848 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:50.191744    5315 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt" podUID="7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:50.231770 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:50.231715    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s" podUID="155e4b39-1877-4bd9-8541-5d6bd3aa0a8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Nov 03 18:06:50.242748 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:50.242697    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7" podUID="69760dba-faf9-4c08-895c-ad70fe95fd49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:50.245050 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:50.245010    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l" podUID="114dcf54-e4eb-4294-a77b-604fe6d57d48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:50.464697 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:50.443755    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c" podUID="8ac59539-8ac1-466e-8d1b-b2be4c42c708" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:50.468361 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:50.468318    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef-kube-api-access-9fftm" pod "1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef" (UID: "1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef"). InnerVolumeSpecName "kube-api-access-9fftm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:06:50.523658 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:50.493468    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q" podUID="c974b0f1-e00b-4e4a-b193-df769068983d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 18:06:50.523658 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:50.503018    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7" podUID="09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Nov 03 18:06:50.523658 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:50.503280    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr" podUID="df8e0b71-e876-4cea-a7ae-3532c887c660" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 18:06:50.621216 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:50.621172    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7" podUID="33725a51-6459-4f15-a866-e9dee41a9732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 03 18:06:50.688840 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:50.656134    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:50.688840 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:50.686426    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:50.688840 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:50.685774    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv" podUID="f6d706cb-3aa5-419c-8f6c-29347bb788b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 18:06:50.829631 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:50.807916    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:06:50.829631 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:50.807982    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p" podUID="79050b41-0f3b-4fc4-a8e8-50316359ac33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:50.994761 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:50.685021    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr" podUID="08159a02-fc2e-4138-824d-be5ba3e141d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:51.144977 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:06:51.029486    5315 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.809s"
Nov 03 18:06:51.144977 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.132732    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7"
Nov 03 18:06:51.144977 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.132881    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr"
Nov 03 18:06:51.144977 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.132929    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr"
Nov 03 18:06:51.144977 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.132964    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt"
Nov 03 18:06:51.144977 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.132996    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s"
Nov 03 18:06:51.144977 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.133029    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q"
Nov 03 18:06:51.144977 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.133058    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt"
Nov 03 18:06:51.144977 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.142365    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7"
Nov 03 18:06:51.150804 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.150677    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr"
Nov 03 18:06:51.151275 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.151254    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c"
Nov 03 18:06:51.175046 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.174934    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3920dd9f-vrgft"]
Nov 03 18:06:51.176230 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.176201    5315 cpu_manager.go:414] "containerMap: removing container" podUID="1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef" containerName="revoke-keys"
Nov 03 18:06:51.176392 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.176378    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef" containerName="revoke-keys"
Nov 03 18:06:51.176673 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.176657    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef" containerName="revoke-keys"
Nov 03 18:06:51.190016 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.189978    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3920dd9f-vrgft"]
Nov 03 18:06:51.190243 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.190229    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdf4f8f0-nr8cs"]
Nov 03 18:06:51.190465 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.190445    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3920dd9f-vrgft"
Nov 03 18:06:51.216629 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.200195    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdf4f8f0-nr8cs"]
Nov 03 18:06:51.229948 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.217110    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7"
Nov 03 18:06:51.263700 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.261371    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fftm\" (UniqueName: \"kubernetes.io/projected/1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef-kube-api-access-9fftm\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:51.269806 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.264634    5315 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:06:51.269806 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.264686    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp" podUID="3cf05bbd-6605-4cd2-aa94-85cbb46223aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:51.301709 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.301681    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaaeec56-qxbj2"
Nov 03 18:06:51.313066 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.313032    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="863d7747-9354-4c32-92de-2d4c26148682" path="/var/lib/kubelet/pods/863d7747-9354-4c32-92de-2d4c26148682/volumes"
Nov 03 18:06:51.313746 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.313720    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaaeec56-qxbj2" event={"ID":"1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef","Type":"ContainerDied","Data":"950d6e871455ef6bec279e0b8a24bfee7e52cc038360213b65b3c8add5467844"}
Nov 03 18:06:51.331415 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.329328    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="950d6e871455ef6bec279e0b8a24bfee7e52cc038360213b65b3c8add5467844"
Nov 03 18:06:51.331415 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.329410    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr"
Nov 03 18:06:51.331415 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.329461    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q"
Nov 03 18:06:51.331415 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.329499    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s"
Nov 03 18:06:51.331415 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.329534    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv"
Nov 03 18:06:51.331415 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.329569    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l"
Nov 03 18:06:51.331415 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.329601    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7"
Nov 03 18:06:51.338182 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.332326    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9ad588-jp7t8"
Nov 03 18:06:51.363116 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.363079    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3920dd9f-vrgft\" (UID: \"5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3920dd9f-vrgft"
Nov 03 18:06:51.363747 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.363718    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3920dd9f-vrgft\" (UID: \"5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3920dd9f-vrgft"
Nov 03 18:06:51.363871 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.363779    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g56l2\" (UniqueName: \"kubernetes.io/projected/5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f-kube-api-access-g56l2\") pod \"maas-api-revoke-sub-3920dd9f-vrgft\" (UID: \"5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3920dd9f-vrgft"
Nov 03 18:06:51.408510 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:06:51.408475    5315 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fb39614_be67_47c2_a0a6_dd8d1bd6b2ef.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 18:06:51.465398 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.465366    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/051dcdb7-a6c4-4167-b814-8b16bb04fb4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/051dcdb7-a6c4-4167-b814-8b16bb04fb4e-maas-api-xks-ca\") pod \"051dcdb7-a6c4-4167-b814-8b16bb04fb4e\" (UID: \"051dcdb7-a6c4-4167-b814-8b16bb04fb4e\") "
Nov 03 18:06:51.465896 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.465419    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/051dcdb7-a6c4-4167-b814-8b16bb04fb4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/051dcdb7-a6c4-4167-b814-8b16bb04fb4e-maas-api-openshift-ca\") pod \"051dcdb7-a6c4-4167-b814-8b16bb04fb4e\" (UID: \"051dcdb7-a6c4-4167-b814-8b16bb04fb4e\") "
Nov 03 18:06:51.465896 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.465493    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/051dcdb7-a6c4-4167-b814-8b16bb04fb4e-kube-api-access-g72ft\" (UniqueName: \"kubernetes.io/projected/051dcdb7-a6c4-4167-b814-8b16bb04fb4e-kube-api-access-g72ft\") pod \"051dcdb7-a6c4-4167-b814-8b16bb04fb4e\" (UID: \"051dcdb7-a6c4-4167-b814-8b16bb04fb4e\") "
Nov 03 18:06:51.466023 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.465887    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/051dcdb7-a6c4-4167-b814-8b16bb04fb4e-maas-api-openshift-ca" pod "051dcdb7-a6c4-4167-b814-8b16bb04fb4e" (UID: "051dcdb7-a6c4-4167-b814-8b16bb04fb4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:06:51.466076 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.466030    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3920dd9f-vrgft\" (UID: \"5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3920dd9f-vrgft"
Nov 03 18:06:51.466191 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.466170    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3920dd9f-vrgft\" (UID: \"5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3920dd9f-vrgft"
Nov 03 18:06:51.466280 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.466262    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g56l2\" (UniqueName: \"kubernetes.io/projected/5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f-kube-api-access-g56l2\") pod \"maas-api-revoke-sub-3920dd9f-vrgft\" (UID: \"5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3920dd9f-vrgft"
Nov 03 18:06:51.467852 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.467825    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3920dd9f-vrgft\" (UID: \"5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3920dd9f-vrgft"
Nov 03 18:06:51.468116 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.468029    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/051dcdb7-a6c4-4167-b814-8b16bb04fb4e-maas-api-xks-ca" pod "051dcdb7-a6c4-4167-b814-8b16bb04fb4e" (UID: "051dcdb7-a6c4-4167-b814-8b16bb04fb4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:06:51.478355 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.468582    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/051dcdb7-a6c4-4167-b814-8b16bb04fb4e-kube-api-access-g72ft" pod "051dcdb7-a6c4-4167-b814-8b16bb04fb4e" (UID: "051dcdb7-a6c4-4167-b814-8b16bb04fb4e"). InnerVolumeSpecName "kube-api-access-g72ft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:06:51.478355 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.468307    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/051dcdb7-a6c4-4167-b814-8b16bb04fb4e-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:51.478559 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.478461    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3920dd9f-vrgft\" (UID: \"5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3920dd9f-vrgft"
Nov 03 18:06:51.525841 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.525793    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g56l2\" (UniqueName: \"kubernetes.io/projected/5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f-kube-api-access-g56l2\") pod \"maas-api-revoke-sub-3920dd9f-vrgft\" (UID: \"5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3920dd9f-vrgft"
Nov 03 18:06:51.626824 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.619949    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/051dcdb7-a6c4-4167-b814-8b16bb04fb4e-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:51.626824 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.619996    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g72ft\" (UniqueName: \"kubernetes.io/projected/051dcdb7-a6c4-4167-b814-8b16bb04fb4e-kube-api-access-g72ft\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:51.669230 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:51.668926    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3920dd9f-vrgft"
Nov 03 18:06:52.087237 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:52.087200    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3920dd9f-vrgft"]
Nov 03 18:06:52.283674 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:52.283638    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3920dd9f-vrgft" event={"ID":"5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f","Type":"ContainerStarted","Data":"09b4c3409c0fd5d6095c5382bfd45f99a0ff5200245131a9a75e780151b7d40a"}
Nov 03 18:06:52.285180 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:52.285156    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9ad588-jp7t8"
Nov 03 18:06:52.285312 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:52.285177    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9ad588-jp7t8" event={"ID":"051dcdb7-a6c4-4167-b814-8b16bb04fb4e","Type":"ContainerDied","Data":"dafdd6fdb06cc31a3d278a3634a815ca1e29e0b87907998fd35b2eb598bfd113"}
Nov 03 18:06:52.285312 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:52.285219    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dafdd6fdb06cc31a3d278a3634a815ca1e29e0b87907998fd35b2eb598bfd113"
Nov 03 18:06:53.294403 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:53.294368    5315 generic.go:356] "Generic (PLEG): container finished" podID="5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f" containerID="b71ac0ea402dbd2ed532dc68a2ede773659b36a65cf39c7f70cd9bded6c60d00" exitCode=0
Nov 03 18:06:53.294822 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:53.294457    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3920dd9f-vrgft" event={"ID":"5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f","Type":"ContainerDied","Data":"b71ac0ea402dbd2ed532dc68a2ede773659b36a65cf39c7f70cd9bded6c60d00"}
Nov 03 18:06:54.455966 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:54.455939    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3920dd9f-vrgft"
Nov 03 18:06:54.559939 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:54.559860    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f-kube-api-access-g56l2\" (UniqueName: \"kubernetes.io/projected/5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f-kube-api-access-g56l2\") pod \"5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f\" (UID: \"5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f\") "
Nov 03 18:06:54.560073 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:54.560014    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f-maas-api-xks-ca\") pod \"5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f\" (UID: \"5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f\") "
Nov 03 18:06:54.560110 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:54.560072    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f-maas-api-openshift-ca\") pod \"5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f\" (UID: \"5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f\") "
Nov 03 18:06:54.560540 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:54.560493    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f-maas-api-openshift-ca" pod "5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f" (UID: "5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:06:54.562077 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:54.562048    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f-kube-api-access-g56l2" pod "5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f" (UID: "5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f"). InnerVolumeSpecName "kube-api-access-g56l2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:06:54.562177 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:54.562093    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f-maas-api-xks-ca" pod "5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f" (UID: "5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:06:54.585066 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:06:54.585018    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:06:54.661407 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:54.661368    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:54.661407 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:54.661403    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:54.661407 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:54.661413    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g56l2\" (UniqueName: \"kubernetes.io/projected/5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f-kube-api-access-g56l2\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:54.925818 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:54.925733    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99aae353-5k9vd"]
Nov 03 18:06:54.926371 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:54.926357    5315 cpu_manager.go:414] "containerMap: removing container" podUID="051dcdb7-a6c4-4167-b814-8b16bb04fb4e" containerName="revoke-keys"
Nov 03 18:06:54.926416 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:54.926373    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="051dcdb7-a6c4-4167-b814-8b16bb04fb4e" containerName="revoke-keys"
Nov 03 18:06:54.926416 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:54.926381    5315 cpu_manager.go:414] "containerMap: removing container" podUID="5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f" containerName="revoke-keys"
Nov 03 18:06:54.926416 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:54.926387    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f" containerName="revoke-keys"
Nov 03 18:06:54.926511 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:54.926501    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="051dcdb7-a6c4-4167-b814-8b16bb04fb4e" containerName="revoke-keys"
Nov 03 18:06:54.926546 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:54.926512    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f" containerName="revoke-keys"
Nov 03 18:06:54.930388 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:54.930364    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99aae353-5k9vd"
Nov 03 18:06:54.990909 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:54.990876    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99aae353-5k9vd"]
Nov 03 18:06:55.093021 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:55.068202    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55533f46-6666-4424-9d95-9187445ade85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99aae353-5k9vd\" (UID: \"55533f46-6666-4424-9d95-9187445ade85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99aae353-5k9vd"
Nov 03 18:06:55.093021 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:55.090876    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55533f46-6666-4424-9d95-9187445ade85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99aae353-5k9vd\" (UID: \"55533f46-6666-4424-9d95-9187445ade85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99aae353-5k9vd"
Nov 03 18:06:55.093021 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:55.090928    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lzdl\" (UniqueName: \"kubernetes.io/projected/55533f46-6666-4424-9d95-9187445ade85-kube-api-access-5lzdl\") pod \"maas-api-revoke-sub-99aae353-5k9vd\" (UID: \"55533f46-6666-4424-9d95-9187445ade85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99aae353-5k9vd"
Nov 03 18:06:55.205169 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:55.204450    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55533f46-6666-4424-9d95-9187445ade85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99aae353-5k9vd\" (UID: \"55533f46-6666-4424-9d95-9187445ade85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99aae353-5k9vd"
Nov 03 18:06:55.205673 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:55.205636    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55533f46-6666-4424-9d95-9187445ade85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99aae353-5k9vd\" (UID: \"55533f46-6666-4424-9d95-9187445ade85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99aae353-5k9vd"
Nov 03 18:06:55.205986 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:55.205891    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lzdl\" (UniqueName: \"kubernetes.io/projected/55533f46-6666-4424-9d95-9187445ade85-kube-api-access-5lzdl\") pod \"maas-api-revoke-sub-99aae353-5k9vd\" (UID: \"55533f46-6666-4424-9d95-9187445ade85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99aae353-5k9vd"
Nov 03 18:06:55.210316 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:55.207193    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55533f46-6666-4424-9d95-9187445ade85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99aae353-5k9vd\" (UID: \"55533f46-6666-4424-9d95-9187445ade85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99aae353-5k9vd"
Nov 03 18:06:55.237983 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:55.231355    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55533f46-6666-4424-9d95-9187445ade85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99aae353-5k9vd\" (UID: \"55533f46-6666-4424-9d95-9187445ade85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99aae353-5k9vd"
Nov 03 18:06:55.263525 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:55.263496    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lzdl\" (UniqueName: \"kubernetes.io/projected/55533f46-6666-4424-9d95-9187445ade85-kube-api-access-5lzdl\") pod \"maas-api-revoke-sub-99aae353-5k9vd\" (UID: \"55533f46-6666-4424-9d95-9187445ade85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99aae353-5k9vd"
Nov 03 18:06:55.312942 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:55.312905    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3920dd9f-vrgft" event={"ID":"5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f","Type":"ContainerDied","Data":"09b4c3409c0fd5d6095c5382bfd45f99a0ff5200245131a9a75e780151b7d40a"}
Nov 03 18:06:55.313168 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:55.312967    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09b4c3409c0fd5d6095c5382bfd45f99a0ff5200245131a9a75e780151b7d40a"
Nov 03 18:06:55.313168 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:55.312922    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3920dd9f-vrgft"
Nov 03 18:06:55.313509 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:55.313487    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-86101f5d-78fb6dfcd7-4hcbt"]
Nov 03 18:06:55.318339 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:55.318319    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-86101f5d-78fb6dfcd7-4hcbt"
Nov 03 18:06:55.366780 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:55.366755    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-86101f5d\""
Nov 03 18:06:55.366951 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:55.366854    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-86101f5d-dockercfg-rfz8r\""
Nov 03 18:06:55.378941 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:55.378914    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-86101f5d-78fb6dfcd7-4hcbt"]
Nov 03 18:06:55.415304 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:55.415265    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3b1770a-983e-4a1f-8476-c37d1f5fe492-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-86101f5d-78fb6dfcd7-4hcbt\" (UID: \"e3b1770a-983e-4a1f-8476-c37d1f5fe492\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-86101f5d-78fb6dfcd7-4hcbt"
Nov 03 18:06:55.415624 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:55.415579    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5tht\" (UniqueName: \"kubernetes.io/projected/e3b1770a-983e-4a1f-8476-c37d1f5fe492-kube-api-access-g5tht\") pod \"payload-pre-processing-e2e-migrate-86101f5d-78fb6dfcd7-4hcbt\" (UID: \"e3b1770a-983e-4a1f-8476-c37d1f5fe492\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-86101f5d-78fb6dfcd7-4hcbt"
Nov 03 18:06:55.628544 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:55.614347    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99aae353-5k9vd"
Nov 03 18:06:55.642020 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:55.629395    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5tht\" (UniqueName: \"kubernetes.io/projected/e3b1770a-983e-4a1f-8476-c37d1f5fe492-kube-api-access-g5tht\") pod \"payload-pre-processing-e2e-migrate-86101f5d-78fb6dfcd7-4hcbt\" (UID: \"e3b1770a-983e-4a1f-8476-c37d1f5fe492\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-86101f5d-78fb6dfcd7-4hcbt"
Nov 03 18:06:55.646721 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:55.646682    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3b1770a-983e-4a1f-8476-c37d1f5fe492-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-86101f5d-78fb6dfcd7-4hcbt\" (UID: \"e3b1770a-983e-4a1f-8476-c37d1f5fe492\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-86101f5d-78fb6dfcd7-4hcbt"
Nov 03 18:06:55.706537 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:55.706327    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3b1770a-983e-4a1f-8476-c37d1f5fe492-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-86101f5d-78fb6dfcd7-4hcbt\" (UID: \"e3b1770a-983e-4a1f-8476-c37d1f5fe492\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-86101f5d-78fb6dfcd7-4hcbt"
Nov 03 18:06:55.764936 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:55.746657    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5tht\" (UniqueName: \"kubernetes.io/projected/e3b1770a-983e-4a1f-8476-c37d1f5fe492-kube-api-access-g5tht\") pod \"payload-pre-processing-e2e-migrate-86101f5d-78fb6dfcd7-4hcbt\" (UID: \"e3b1770a-983e-4a1f-8476-c37d1f5fe492\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-86101f5d-78fb6dfcd7-4hcbt"
Nov 03 18:06:55.932495 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:55.932407    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-86101f5d-78fb6dfcd7-4hcbt"
Nov 03 18:06:56.147381 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:56.147349    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99aae353-5k9vd"]
Nov 03 18:06:56.151469 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:06:56.151439    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55533f46_6666_4424_9d95_9187445ade85.slice/crio-abce6ee2d029077eeb50553f30964f679efd0911785d4e4d03888b1a45db9529 WatchSource:0}: Error finding container abce6ee2d029077eeb50553f30964f679efd0911785d4e4d03888b1a45db9529: Status 404 returned error can't find the container with id abce6ee2d029077eeb50553f30964f679efd0911785d4e4d03888b1a45db9529
Nov 03 18:06:56.309426 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:56.309394    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-86101f5d-78fb6dfcd7-4hcbt"]
Nov 03 18:06:56.314828 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:06:56.314791    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3b1770a_983e_4a1f_8476_c37d1f5fe492.slice/crio-1b4c29c66a51442e3f5ae37aa69962e64f1ada34f714b5f4ae940bd19f92a2ea WatchSource:0}: Error finding container 1b4c29c66a51442e3f5ae37aa69962e64f1ada34f714b5f4ae940bd19f92a2ea: Status 404 returned error can't find the container with id 1b4c29c66a51442e3f5ae37aa69962e64f1ada34f714b5f4ae940bd19f92a2ea
Nov 03 18:06:56.322711 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:56.322682    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99aae353-5k9vd" event={"ID":"55533f46-6666-4424-9d95-9187445ade85","Type":"ContainerStarted","Data":"abce6ee2d029077eeb50553f30964f679efd0911785d4e4d03888b1a45db9529"}
Nov 03 18:06:57.332049 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:57.331932    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-86101f5d-78fb6dfcd7-4hcbt" event={"ID":"e3b1770a-983e-4a1f-8476-c37d1f5fe492","Type":"ContainerStarted","Data":"960f0a4c0b29d0fc75234db263642b653380ab56a05a7739e30e42e48c99a083"}
Nov 03 18:06:57.332049 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:57.332005    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-86101f5d-78fb6dfcd7-4hcbt" event={"ID":"e3b1770a-983e-4a1f-8476-c37d1f5fe492","Type":"ContainerStarted","Data":"1b4c29c66a51442e3f5ae37aa69962e64f1ada34f714b5f4ae940bd19f92a2ea"}
Nov 03 18:06:57.332049 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:57.332034    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-86101f5d-78fb6dfcd7-4hcbt"
Nov 03 18:06:57.333836 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:57.333806    5315 generic.go:356] "Generic (PLEG): container finished" podID="55533f46-6666-4424-9d95-9187445ade85" containerID="ef7b79738d942825e18db8e65849fea2d35213b745c0876ef7ddbae73baede31" exitCode=0
Nov 03 18:06:57.333947 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:57.333897    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99aae353-5k9vd" event={"ID":"55533f46-6666-4424-9d95-9187445ade85","Type":"ContainerDied","Data":"ef7b79738d942825e18db8e65849fea2d35213b745c0876ef7ddbae73baede31"}
Nov 03 18:06:57.454628 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:57.454567    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-86101f5d-78fb6dfcd7-4hcbt" podStartSLOduration=2.454553852 podStartE2EDuration="2.454553852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:06:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:06:57.452340349 +0000 UTC m=+1256.468538417" watchObservedRunningTime="2026-11-03 18:06:57.454553852 +0000 UTC m=+1256.470751920"
Nov 03 18:06:58.154413 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:58.154373    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q"]
Nov 03 18:06:58.159486 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:58.159460    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q"
Nov 03 18:06:58.226633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:58.222757    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-248f8ee3-openshift-default-dockercfg-r22qh\""
Nov 03 18:06:58.401637 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:58.397745    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11671eb9-4824-4e89-9692-b5be9038c549-istio-envoy\") pod \"e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q\" (UID: \"11671eb9-4824-4e89-9692-b5be9038c549\") " pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q"
Nov 03 18:06:58.401637 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:58.397882    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11671eb9-4824-4e89-9692-b5be9038c549-istio-token\") pod \"e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q\" (UID: \"11671eb9-4824-4e89-9692-b5be9038c549\") " pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q"
Nov 03 18:06:58.401637 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:58.397946    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11671eb9-4824-4e89-9692-b5be9038c549-istio-data\") pod \"e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q\" (UID: \"11671eb9-4824-4e89-9692-b5be9038c549\") " pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q"
Nov 03 18:06:58.401637 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:58.398012    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11671eb9-4824-4e89-9692-b5be9038c549-workload-socket\") pod \"e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q\" (UID: \"11671eb9-4824-4e89-9692-b5be9038c549\") " pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q"
Nov 03 18:06:58.401637 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:58.398108    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjt66\" (UniqueName: \"kubernetes.io/projected/11671eb9-4824-4e89-9692-b5be9038c549-kube-api-access-qjt66\") pod \"e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q\" (UID: \"11671eb9-4824-4e89-9692-b5be9038c549\") " pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q"
Nov 03 18:06:58.401637 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:58.398141    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11671eb9-4824-4e89-9692-b5be9038c549-credential-socket\") pod \"e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q\" (UID: \"11671eb9-4824-4e89-9692-b5be9038c549\") " pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q"
Nov 03 18:06:58.401637 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:58.398176    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11671eb9-4824-4e89-9692-b5be9038c549-workload-certs\") pod \"e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q\" (UID: \"11671eb9-4824-4e89-9692-b5be9038c549\") " pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q"
Nov 03 18:06:58.401637 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:58.398232    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11671eb9-4824-4e89-9692-b5be9038c549-istiod-ca-cert\") pod \"e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q\" (UID: \"11671eb9-4824-4e89-9692-b5be9038c549\") " pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q"
Nov 03 18:06:58.401637 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:58.398360    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11671eb9-4824-4e89-9692-b5be9038c549-istio-podinfo\") pod \"e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q\" (UID: \"11671eb9-4824-4e89-9692-b5be9038c549\") " pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q"
Nov 03 18:06:58.481464 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:58.481034    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q"]
Nov 03 18:06:58.579065 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:58.516018    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11671eb9-4824-4e89-9692-b5be9038c549-workload-socket\") pod \"e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q\" (UID: \"11671eb9-4824-4e89-9692-b5be9038c549\") " pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q"
Nov 03 18:06:58.579065 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:58.516159    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjt66\" (UniqueName: \"kubernetes.io/projected/11671eb9-4824-4e89-9692-b5be9038c549-kube-api-access-qjt66\") pod \"e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q\" (UID: \"11671eb9-4824-4e89-9692-b5be9038c549\") " pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q"
Nov 03 18:06:58.579065 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:58.516197    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11671eb9-4824-4e89-9692-b5be9038c549-credential-socket\") pod \"e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q\" (UID: \"11671eb9-4824-4e89-9692-b5be9038c549\") " pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q"
Nov 03 18:06:58.579065 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:58.516227    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11671eb9-4824-4e89-9692-b5be9038c549-workload-certs\") pod \"e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q\" (UID: \"11671eb9-4824-4e89-9692-b5be9038c549\") " pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q"
Nov 03 18:06:58.579065 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:58.516282    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11671eb9-4824-4e89-9692-b5be9038c549-istiod-ca-cert\") pod \"e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q\" (UID: \"11671eb9-4824-4e89-9692-b5be9038c549\") " pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q"
Nov 03 18:06:58.579065 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:58.516396    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11671eb9-4824-4e89-9692-b5be9038c549-istio-podinfo\") pod \"e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q\" (UID: \"11671eb9-4824-4e89-9692-b5be9038c549\") " pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q"
Nov 03 18:06:58.579065 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:58.516435    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11671eb9-4824-4e89-9692-b5be9038c549-istio-envoy\") pod \"e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q\" (UID: \"11671eb9-4824-4e89-9692-b5be9038c549\") " pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q"
Nov 03 18:06:58.579065 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:58.516558    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11671eb9-4824-4e89-9692-b5be9038c549-istio-token\") pod \"e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q\" (UID: \"11671eb9-4824-4e89-9692-b5be9038c549\") " pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q"
Nov 03 18:06:58.579065 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:58.516644    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11671eb9-4824-4e89-9692-b5be9038c549-istio-data\") pod \"e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q\" (UID: \"11671eb9-4824-4e89-9692-b5be9038c549\") " pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q"
Nov 03 18:06:58.579065 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:58.524010    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11671eb9-4824-4e89-9692-b5be9038c549-workload-certs\") pod \"e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q\" (UID: \"11671eb9-4824-4e89-9692-b5be9038c549\") " pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q"
Nov 03 18:06:58.579065 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:58.559983    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11671eb9-4824-4e89-9692-b5be9038c549-istio-data\") pod \"e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q\" (UID: \"11671eb9-4824-4e89-9692-b5be9038c549\") " pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q"
Nov 03 18:06:58.579065 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:58.560586    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11671eb9-4824-4e89-9692-b5be9038c549-workload-socket\") pod \"e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q\" (UID: \"11671eb9-4824-4e89-9692-b5be9038c549\") " pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q"
Nov 03 18:06:58.579065 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:58.561327    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11671eb9-4824-4e89-9692-b5be9038c549-credential-socket\") pod \"e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q\" (UID: \"11671eb9-4824-4e89-9692-b5be9038c549\") " pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q"
Nov 03 18:06:58.579065 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:58.564318    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11671eb9-4824-4e89-9692-b5be9038c549-istiod-ca-cert\") pod \"e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q\" (UID: \"11671eb9-4824-4e89-9692-b5be9038c549\") " pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q"
Nov 03 18:06:58.621857 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:58.613743    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11671eb9-4824-4e89-9692-b5be9038c549-istio-envoy\") pod \"e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q\" (UID: \"11671eb9-4824-4e89-9692-b5be9038c549\") " pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q"
Nov 03 18:06:58.621857 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:58.614945    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11671eb9-4824-4e89-9692-b5be9038c549-istio-podinfo\") pod \"e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q\" (UID: \"11671eb9-4824-4e89-9692-b5be9038c549\") " pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q"
Nov 03 18:06:58.628594 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:58.625088    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjt66\" (UniqueName: \"kubernetes.io/projected/11671eb9-4824-4e89-9692-b5be9038c549-kube-api-access-qjt66\") pod \"e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q\" (UID: \"11671eb9-4824-4e89-9692-b5be9038c549\") " pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q"
Nov 03 18:06:58.631560 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:58.630238    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11671eb9-4824-4e89-9692-b5be9038c549-istio-token\") pod \"e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q\" (UID: \"11671eb9-4824-4e89-9692-b5be9038c549\") " pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q"
Nov 03 18:06:58.792742 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:58.779279    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q"
Nov 03 18:06:58.889004 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:58.888970    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99aae353-5k9vd"
Nov 03 18:06:58.972962 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:58.972925    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55533f46-6666-4424-9d95-9187445ade85-kube-api-access-5lzdl\" (UniqueName: \"kubernetes.io/projected/55533f46-6666-4424-9d95-9187445ade85-kube-api-access-5lzdl\") pod \"55533f46-6666-4424-9d95-9187445ade85\" (UID: \"55533f46-6666-4424-9d95-9187445ade85\") "
Nov 03 18:06:58.973116 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:58.973035    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55533f46-6666-4424-9d95-9187445ade85-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55533f46-6666-4424-9d95-9187445ade85-maas-api-xks-ca\") pod \"55533f46-6666-4424-9d95-9187445ade85\" (UID: \"55533f46-6666-4424-9d95-9187445ade85\") "
Nov 03 18:06:58.973116 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:58.973081    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55533f46-6666-4424-9d95-9187445ade85-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55533f46-6666-4424-9d95-9187445ade85-maas-api-openshift-ca\") pod \"55533f46-6666-4424-9d95-9187445ade85\" (UID: \"55533f46-6666-4424-9d95-9187445ade85\") "
Nov 03 18:06:58.973508 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:58.973485    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55533f46-6666-4424-9d95-9187445ade85-maas-api-openshift-ca" pod "55533f46-6666-4424-9d95-9187445ade85" (UID: "55533f46-6666-4424-9d95-9187445ade85"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:06:58.973808 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:58.973785    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55533f46-6666-4424-9d95-9187445ade85-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:58.975192 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:58.975167    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55533f46-6666-4424-9d95-9187445ade85-maas-api-xks-ca" pod "55533f46-6666-4424-9d95-9187445ade85" (UID: "55533f46-6666-4424-9d95-9187445ade85"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:06:58.975268 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:58.975195    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55533f46-6666-4424-9d95-9187445ade85-kube-api-access-5lzdl" pod "55533f46-6666-4424-9d95-9187445ade85" (UID: "55533f46-6666-4424-9d95-9187445ade85"). InnerVolumeSpecName "kube-api-access-5lzdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:06:59.075446 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:59.075416    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lzdl\" (UniqueName: \"kubernetes.io/projected/55533f46-6666-4424-9d95-9187445ade85-kube-api-access-5lzdl\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:59.075446 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:59.075444    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55533f46-6666-4424-9d95-9187445ade85-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:59.241410 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:59.241366    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q"]
Nov 03 18:06:59.246060 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:06:59.246029    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11671eb9_4824_4e89_9692_b5be9038c549.slice/crio-767003902d0a4ff4f427e48e0cd0cd3df919c719b30d41d6fe286c36ae43fb9d WatchSource:0}: Error finding container 767003902d0a4ff4f427e48e0cd0cd3df919c719b30d41d6fe286c36ae43fb9d: Status 404 returned error can't find the container with id 767003902d0a4ff4f427e48e0cd0cd3df919c719b30d41d6fe286c36ae43fb9d
Nov 03 18:06:59.249013 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:59.248961    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:06:59.249126 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:59.249062    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:06:59.249126 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:59.249108    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:06:59.629671 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:59.629640    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q" event={"ID":"11671eb9-4824-4e89-9692-b5be9038c549","Type":"ContainerStarted","Data":"31303af85f59ce20708ef7dfcdeebc3fe83cf483a50ab05bf91b7020b28cb973"}
Nov 03 18:06:59.630099 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:59.629701    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q" event={"ID":"11671eb9-4824-4e89-9692-b5be9038c549","Type":"ContainerStarted","Data":"767003902d0a4ff4f427e48e0cd0cd3df919c719b30d41d6fe286c36ae43fb9d"}
Nov 03 18:06:59.631358 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:59.631331    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99aae353-5k9vd" event={"ID":"55533f46-6666-4424-9d95-9187445ade85","Type":"ContainerDied","Data":"abce6ee2d029077eeb50553f30964f679efd0911785d4e4d03888b1a45db9529"}
Nov 03 18:06:59.631469 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:59.631359    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99aae353-5k9vd"
Nov 03 18:06:59.631469 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:59.631376    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abce6ee2d029077eeb50553f30964f679efd0911785d4e4d03888b1a45db9529"
Nov 03 18:06:59.776137 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:59.776035    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q" podStartSLOduration=1.776006948 podStartE2EDuration="1.776006948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:06:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:06:59.754961737 +0000 UTC m=+1258.771159798" watchObservedRunningTime="2026-11-03 18:06:59.776006948 +0000 UTC m=+1258.792205044"
Nov 03 18:06:59.792380 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:06:59.788239    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q"
Nov 03 18:07:00.882384 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:00.867499    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q"
Nov 03 18:07:01.378131 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:01.378086    5315 scope.go:122] "RemoveContainer" containerID="1487fe3544bdc6c0b8783724d1b2b0f99a977f4026265935b9684424a5f2f79a"
Nov 03 18:07:01.391174 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:01.391124    5315 scope.go:122] "RemoveContainer" containerID="5b9be3dfca82b13962ed31c7dc7f54bc790b899bf8f901c27fe9f769930d450b"
Nov 03 18:07:01.404677 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:01.404647    5315 scope.go:122] "RemoveContainer" containerID="f7c1e246d4a71c9958f4e2b48dc4129e1d606f4d86ab9daf1829a87bb6594594"
Nov 03 18:07:01.416437 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:01.416408    5315 scope.go:122] "RemoveContainer" containerID="52718e14e23520673d90530aed297f1a06ed79ef2b95d32514e05261def7b512"
Nov 03 18:07:01.429861 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:01.429816    5315 scope.go:122] "RemoveContainer" containerID="160693fb5c554625f74c24e63a849d26f7816d3d6f4ab3f3fb1be38025ef210b"
Nov 03 18:07:01.442494 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:01.442460    5315 scope.go:122] "RemoveContainer" containerID="d450a6f5c7cdec09d0f499c4e4b1c34562b6d471544082e953012d3af44ad3e8"
Nov 03 18:07:01.456981 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:01.456957    5315 scope.go:122] "RemoveContainer" containerID="c8058cd11709e2bd49974f2d27a0b154b980ac3032b554f318a70174faed6480"
Nov 03 18:07:01.660569 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:01.660475    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q"
Nov 03 18:07:01.661386 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:01.661367    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q"
Nov 03 18:07:08.622899 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:08.622860    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-86101f5d-78fb6dfcd7-4hcbt"
Nov 03 18:07:09.534083 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:09.534035    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr"]
Nov 03 18:07:09.534323 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:09.534291    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr" podUID="8f5bff5e-4b15-4cde-a3b0-71c9ae666122" containerName="istio-proxy" containerID="cri-o://61bcac3edd0b11569024e70049f166240dc379a0b07b8d6febfe7be692b1e5f7" gracePeriod=30
Nov 03 18:07:12.085836 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:12.085752    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn"]
Nov 03 18:07:12.086432 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:12.086415    5315 cpu_manager.go:414] "containerMap: removing container" podUID="55533f46-6666-4424-9d95-9187445ade85" containerName="revoke-keys"
Nov 03 18:07:12.086474 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:12.086433    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55533f46-6666-4424-9d95-9187445ade85" containerName="revoke-keys"
Nov 03 18:07:12.086573 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:12.086564    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="55533f46-6666-4424-9d95-9187445ade85" containerName="revoke-keys"
Nov 03 18:07:12.095959 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:12.095933    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn"
Nov 03 18:07:12.201007 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:12.200967    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn"]
Nov 03 18:07:12.226068 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:12.226027    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dafbc7c7-538e-4bf6-ab30-970870f21854-istio-data\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn\" (UID: \"dafbc7c7-538e-4bf6-ab30-970870f21854\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn"
Nov 03 18:07:12.226261 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:12.226192    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dafbc7c7-538e-4bf6-ab30-970870f21854-istio-envoy\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn\" (UID: \"dafbc7c7-538e-4bf6-ab30-970870f21854\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn"
Nov 03 18:07:12.226346 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:12.226332    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dafbc7c7-538e-4bf6-ab30-970870f21854-istio-podinfo\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn\" (UID: \"dafbc7c7-538e-4bf6-ab30-970870f21854\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn"
Nov 03 18:07:12.226409 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:12.226353    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngwk4\" (UniqueName: \"kubernetes.io/projected/dafbc7c7-538e-4bf6-ab30-970870f21854-kube-api-access-ngwk4\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn\" (UID: \"dafbc7c7-538e-4bf6-ab30-970870f21854\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn"
Nov 03 18:07:12.226463 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:12.226436    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dafbc7c7-538e-4bf6-ab30-970870f21854-istio-token\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn\" (UID: \"dafbc7c7-538e-4bf6-ab30-970870f21854\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn"
Nov 03 18:07:12.226512 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:12.226480    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafbc7c7-538e-4bf6-ab30-970870f21854-credential-socket\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn\" (UID: \"dafbc7c7-538e-4bf6-ab30-970870f21854\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn"
Nov 03 18:07:12.226560 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:12.226511    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dafbc7c7-538e-4bf6-ab30-970870f21854-workload-certs\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn\" (UID: \"dafbc7c7-538e-4bf6-ab30-970870f21854\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn"
Nov 03 18:07:12.226560 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:12.226545    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafbc7c7-538e-4bf6-ab30-970870f21854-workload-socket\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn\" (UID: \"dafbc7c7-538e-4bf6-ab30-970870f21854\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn"
Nov 03 18:07:12.226688 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:12.226596    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dafbc7c7-538e-4bf6-ab30-970870f21854-istiod-ca-cert\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn\" (UID: \"dafbc7c7-538e-4bf6-ab30-970870f21854\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn"
Nov 03 18:07:12.327861 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:12.327816    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafbc7c7-538e-4bf6-ab30-970870f21854-credential-socket\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn\" (UID: \"dafbc7c7-538e-4bf6-ab30-970870f21854\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn"
Nov 03 18:07:12.328056 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:12.327874    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dafbc7c7-538e-4bf6-ab30-970870f21854-workload-certs\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn\" (UID: \"dafbc7c7-538e-4bf6-ab30-970870f21854\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn"
Nov 03 18:07:12.328056 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:12.327900    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafbc7c7-538e-4bf6-ab30-970870f21854-workload-socket\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn\" (UID: \"dafbc7c7-538e-4bf6-ab30-970870f21854\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn"
Nov 03 18:07:12.328056 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:12.327916    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dafbc7c7-538e-4bf6-ab30-970870f21854-istiod-ca-cert\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn\" (UID: \"dafbc7c7-538e-4bf6-ab30-970870f21854\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn"
Nov 03 18:07:12.328056 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:12.327969    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dafbc7c7-538e-4bf6-ab30-970870f21854-istio-data\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn\" (UID: \"dafbc7c7-538e-4bf6-ab30-970870f21854\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn"
Nov 03 18:07:12.328056 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:12.328031    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dafbc7c7-538e-4bf6-ab30-970870f21854-istio-envoy\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn\" (UID: \"dafbc7c7-538e-4bf6-ab30-970870f21854\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn"
Nov 03 18:07:12.328329 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:12.328176    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dafbc7c7-538e-4bf6-ab30-970870f21854-istio-podinfo\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn\" (UID: \"dafbc7c7-538e-4bf6-ab30-970870f21854\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn"
Nov 03 18:07:12.328329 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:12.328205    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngwk4\" (UniqueName: \"kubernetes.io/projected/dafbc7c7-538e-4bf6-ab30-970870f21854-kube-api-access-ngwk4\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn\" (UID: \"dafbc7c7-538e-4bf6-ab30-970870f21854\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn"
Nov 03 18:07:12.328329 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:12.328274    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dafbc7c7-538e-4bf6-ab30-970870f21854-istio-token\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn\" (UID: \"dafbc7c7-538e-4bf6-ab30-970870f21854\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn"
Nov 03 18:07:12.328492 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:12.328323    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafbc7c7-538e-4bf6-ab30-970870f21854-workload-socket\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn\" (UID: \"dafbc7c7-538e-4bf6-ab30-970870f21854\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn"
Nov 03 18:07:12.328492 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:12.328435    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dafbc7c7-538e-4bf6-ab30-970870f21854-istio-data\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn\" (UID: \"dafbc7c7-538e-4bf6-ab30-970870f21854\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn"
Nov 03 18:07:12.328604 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:12.328551    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafbc7c7-538e-4bf6-ab30-970870f21854-credential-socket\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn\" (UID: \"dafbc7c7-538e-4bf6-ab30-970870f21854\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn"
Nov 03 18:07:12.328804 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:12.328772    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dafbc7c7-538e-4bf6-ab30-970870f21854-istiod-ca-cert\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn\" (UID: \"dafbc7c7-538e-4bf6-ab30-970870f21854\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn"
Nov 03 18:07:12.328924 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:12.328880    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dafbc7c7-538e-4bf6-ab30-970870f21854-workload-certs\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn\" (UID: \"dafbc7c7-538e-4bf6-ab30-970870f21854\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn"
Nov 03 18:07:12.330745 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:12.330721    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dafbc7c7-538e-4bf6-ab30-970870f21854-istio-envoy\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn\" (UID: \"dafbc7c7-538e-4bf6-ab30-970870f21854\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn"
Nov 03 18:07:12.330847 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:12.330760    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dafbc7c7-538e-4bf6-ab30-970870f21854-istio-podinfo\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn\" (UID: \"dafbc7c7-538e-4bf6-ab30-970870f21854\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn"
Nov 03 18:07:12.385742 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:12.385664    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dafbc7c7-538e-4bf6-ab30-970870f21854-istio-token\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn\" (UID: \"dafbc7c7-538e-4bf6-ab30-970870f21854\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn"
Nov 03 18:07:12.385877 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:12.385758    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngwk4\" (UniqueName: \"kubernetes.io/projected/dafbc7c7-538e-4bf6-ab30-970870f21854-kube-api-access-ngwk4\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn\" (UID: \"dafbc7c7-538e-4bf6-ab30-970870f21854\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn"
Nov 03 18:07:12.409372 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:12.409345    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn"
Nov 03 18:07:12.844285 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:12.844244    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn"]
Nov 03 18:07:12.848603 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:07:12.848566    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddafbc7c7_538e_4bf6_ab30_970870f21854.slice/crio-2ec0a319d634e0b08756b134f2e00e9216500a1de1f43e7c4b084db39f7ab4d6 WatchSource:0}: Error finding container 2ec0a319d634e0b08756b134f2e00e9216500a1de1f43e7c4b084db39f7ab4d6: Status 404 returned error can't find the container with id 2ec0a319d634e0b08756b134f2e00e9216500a1de1f43e7c4b084db39f7ab4d6
Nov 03 18:07:12.851391 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:12.851354    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:07:12.851491 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:12.851425    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:07:12.851491 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:12.851457    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:07:13.328276 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.328241    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n"]
Nov 03 18:07:13.333649 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.333583    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n"
Nov 03 18:07:13.466985 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.466942    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e4ca152-3632-485f-94b2-4f430cbdfe4c-istio-data\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n\" (UID: \"1e4ca152-3632-485f-94b2-4f430cbdfe4c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n"
Nov 03 18:07:13.467178 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.467114    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e4ca152-3632-485f-94b2-4f430cbdfe4c-istio-envoy\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n\" (UID: \"1e4ca152-3632-485f-94b2-4f430cbdfe4c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n"
Nov 03 18:07:13.467178 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.467159    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e4ca152-3632-485f-94b2-4f430cbdfe4c-istiod-ca-cert\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n\" (UID: \"1e4ca152-3632-485f-94b2-4f430cbdfe4c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n"
Nov 03 18:07:13.467297 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.467186    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e4ca152-3632-485f-94b2-4f430cbdfe4c-credential-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n\" (UID: \"1e4ca152-3632-485f-94b2-4f430cbdfe4c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n"
Nov 03 18:07:13.467297 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.467256    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e4ca152-3632-485f-94b2-4f430cbdfe4c-istio-token\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n\" (UID: \"1e4ca152-3632-485f-94b2-4f430cbdfe4c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n"
Nov 03 18:07:13.467408 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.467316    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e4ca152-3632-485f-94b2-4f430cbdfe4c-istio-podinfo\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n\" (UID: \"1e4ca152-3632-485f-94b2-4f430cbdfe4c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n"
Nov 03 18:07:13.467408 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.467379    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e4ca152-3632-485f-94b2-4f430cbdfe4c-workload-certs\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n\" (UID: \"1e4ca152-3632-485f-94b2-4f430cbdfe4c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n"
Nov 03 18:07:13.467506 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.467406    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlrnr\" (UniqueName: \"kubernetes.io/projected/1e4ca152-3632-485f-94b2-4f430cbdfe4c-kube-api-access-wlrnr\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n\" (UID: \"1e4ca152-3632-485f-94b2-4f430cbdfe4c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n"
Nov 03 18:07:13.467506 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.467433    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e4ca152-3632-485f-94b2-4f430cbdfe4c-workload-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n\" (UID: \"1e4ca152-3632-485f-94b2-4f430cbdfe4c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n"
Nov 03 18:07:13.470289 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.470253    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n"]
Nov 03 18:07:13.569288 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.569239    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e4ca152-3632-485f-94b2-4f430cbdfe4c-istio-envoy\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n\" (UID: \"1e4ca152-3632-485f-94b2-4f430cbdfe4c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n"
Nov 03 18:07:13.569457 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.569316    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e4ca152-3632-485f-94b2-4f430cbdfe4c-istiod-ca-cert\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n\" (UID: \"1e4ca152-3632-485f-94b2-4f430cbdfe4c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n"
Nov 03 18:07:13.569457 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.569345    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e4ca152-3632-485f-94b2-4f430cbdfe4c-credential-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n\" (UID: \"1e4ca152-3632-485f-94b2-4f430cbdfe4c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n"
Nov 03 18:07:13.569457 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.569406    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e4ca152-3632-485f-94b2-4f430cbdfe4c-istio-token\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n\" (UID: \"1e4ca152-3632-485f-94b2-4f430cbdfe4c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n"
Nov 03 18:07:13.569457 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.569440    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e4ca152-3632-485f-94b2-4f430cbdfe4c-istio-podinfo\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n\" (UID: \"1e4ca152-3632-485f-94b2-4f430cbdfe4c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n"
Nov 03 18:07:13.569702 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.569486    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e4ca152-3632-485f-94b2-4f430cbdfe4c-workload-certs\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n\" (UID: \"1e4ca152-3632-485f-94b2-4f430cbdfe4c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n"
Nov 03 18:07:13.569702 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.569509    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlrnr\" (UniqueName: \"kubernetes.io/projected/1e4ca152-3632-485f-94b2-4f430cbdfe4c-kube-api-access-wlrnr\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n\" (UID: \"1e4ca152-3632-485f-94b2-4f430cbdfe4c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n"
Nov 03 18:07:13.569702 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.569536    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e4ca152-3632-485f-94b2-4f430cbdfe4c-workload-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n\" (UID: \"1e4ca152-3632-485f-94b2-4f430cbdfe4c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n"
Nov 03 18:07:13.569873 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.569696    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e4ca152-3632-485f-94b2-4f430cbdfe4c-istio-data\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n\" (UID: \"1e4ca152-3632-485f-94b2-4f430cbdfe4c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n"
Nov 03 18:07:13.569873 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.569791    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e4ca152-3632-485f-94b2-4f430cbdfe4c-credential-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n\" (UID: \"1e4ca152-3632-485f-94b2-4f430cbdfe4c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n"
Nov 03 18:07:13.570120 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.570097    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e4ca152-3632-485f-94b2-4f430cbdfe4c-workload-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n\" (UID: \"1e4ca152-3632-485f-94b2-4f430cbdfe4c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n"
Nov 03 18:07:13.570234 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.570179    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e4ca152-3632-485f-94b2-4f430cbdfe4c-workload-certs\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n\" (UID: \"1e4ca152-3632-485f-94b2-4f430cbdfe4c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n"
Nov 03 18:07:13.570234 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.570184    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e4ca152-3632-485f-94b2-4f430cbdfe4c-istiod-ca-cert\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n\" (UID: \"1e4ca152-3632-485f-94b2-4f430cbdfe4c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n"
Nov 03 18:07:13.570234 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.570222    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e4ca152-3632-485f-94b2-4f430cbdfe4c-istio-data\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n\" (UID: \"1e4ca152-3632-485f-94b2-4f430cbdfe4c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n"
Nov 03 18:07:13.571720 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.571695    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e4ca152-3632-485f-94b2-4f430cbdfe4c-istio-envoy\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n\" (UID: \"1e4ca152-3632-485f-94b2-4f430cbdfe4c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n"
Nov 03 18:07:13.572129 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.572111    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e4ca152-3632-485f-94b2-4f430cbdfe4c-istio-podinfo\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n\" (UID: \"1e4ca152-3632-485f-94b2-4f430cbdfe4c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n"
Nov 03 18:07:13.651061 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.650979    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e4ca152-3632-485f-94b2-4f430cbdfe4c-istio-token\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n\" (UID: \"1e4ca152-3632-485f-94b2-4f430cbdfe4c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n"
Nov 03 18:07:13.651205 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.651092    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlrnr\" (UniqueName: \"kubernetes.io/projected/1e4ca152-3632-485f-94b2-4f430cbdfe4c-kube-api-access-wlrnr\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n\" (UID: \"1e4ca152-3632-485f-94b2-4f430cbdfe4c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n"
Nov 03 18:07:13.663735 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.663709    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n"
Nov 03 18:07:13.791935 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.791891    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn" event={"ID":"dafbc7c7-538e-4bf6-ab30-970870f21854","Type":"ContainerStarted","Data":"131139a80d6fb7ad51d73854a2036d32c7d660b479bc2c84808e89e370b8592a"}
Nov 03 18:07:13.792134 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.791965    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn" event={"ID":"dafbc7c7-538e-4bf6-ab30-970870f21854","Type":"ContainerStarted","Data":"2ec0a319d634e0b08756b134f2e00e9216500a1de1f43e7c4b084db39f7ab4d6"}
Nov 03 18:07:13.812361 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.812320    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82"]
Nov 03 18:07:13.818552 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.818523    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82"
Nov 03 18:07:13.856651 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.851484    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t"]
Nov 03 18:07:13.865484 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.863560    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t"
Nov 03 18:07:13.888863 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.885362    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82"]
Nov 03 18:07:13.905802 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.905715    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t"]
Nov 03 18:07:13.914665 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.914382    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn" podStartSLOduration=1.914362794 podStartE2EDuration="1.914362794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:07:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:07:13.911445356 +0000 UTC m=+1272.927643448" watchObservedRunningTime="2026-11-03 18:07:13.914362794 +0000 UTC m=+1272.930560864"
Nov 03 18:07:13.987746 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.987711    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klgts\" (UniqueName: \"kubernetes.io/projected/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-kube-api-access-klgts\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82\" (UID: \"04fd7ef5-cf3c-4ed7-963d-e03294eaf327\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82"
Nov 03 18:07:13.987922 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.987762    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b453729-cac0-4839-bdfa-0da04e59c1bc-credential-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t\" (UID: \"2b453729-cac0-4839-bdfa-0da04e59c1bc\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t"
Nov 03 18:07:13.987922 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.987792    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f55pm\" (UniqueName: \"kubernetes.io/projected/2b453729-cac0-4839-bdfa-0da04e59c1bc-kube-api-access-f55pm\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t\" (UID: \"2b453729-cac0-4839-bdfa-0da04e59c1bc\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t"
Nov 03 18:07:13.987922 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.987881    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-istio-data\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82\" (UID: \"04fd7ef5-cf3c-4ed7-963d-e03294eaf327\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82"
Nov 03 18:07:13.988029 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.987943    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-istiod-ca-cert\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82\" (UID: \"04fd7ef5-cf3c-4ed7-963d-e03294eaf327\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82"
Nov 03 18:07:13.988029 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.987964    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b453729-cac0-4839-bdfa-0da04e59c1bc-istio-podinfo\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t\" (UID: \"2b453729-cac0-4839-bdfa-0da04e59c1bc\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t"
Nov 03 18:07:13.988029 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.988005    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-workload-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82\" (UID: \"04fd7ef5-cf3c-4ed7-963d-e03294eaf327\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82"
Nov 03 18:07:13.988136 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.988110    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-workload-certs\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82\" (UID: \"04fd7ef5-cf3c-4ed7-963d-e03294eaf327\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82"
Nov 03 18:07:13.988268 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.988253    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-istio-envoy\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82\" (UID: \"04fd7ef5-cf3c-4ed7-963d-e03294eaf327\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82"
Nov 03 18:07:13.988322 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.988288    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b453729-cac0-4839-bdfa-0da04e59c1bc-istio-token\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t\" (UID: \"2b453729-cac0-4839-bdfa-0da04e59c1bc\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t"
Nov 03 18:07:13.988322 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.988312    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-istio-podinfo\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82\" (UID: \"04fd7ef5-cf3c-4ed7-963d-e03294eaf327\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82"
Nov 03 18:07:13.988414 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.988357    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-credential-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82\" (UID: \"04fd7ef5-cf3c-4ed7-963d-e03294eaf327\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82"
Nov 03 18:07:13.988414 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.988397    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b453729-cac0-4839-bdfa-0da04e59c1bc-istiod-ca-cert\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t\" (UID: \"2b453729-cac0-4839-bdfa-0da04e59c1bc\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t"
Nov 03 18:07:13.988479 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.988441    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-istio-token\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82\" (UID: \"04fd7ef5-cf3c-4ed7-963d-e03294eaf327\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82"
Nov 03 18:07:13.988479 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.988469    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b453729-cac0-4839-bdfa-0da04e59c1bc-istio-data\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t\" (UID: \"2b453729-cac0-4839-bdfa-0da04e59c1bc\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t"
Nov 03 18:07:13.988554 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.988542    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b453729-cac0-4839-bdfa-0da04e59c1bc-workload-certs\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t\" (UID: \"2b453729-cac0-4839-bdfa-0da04e59c1bc\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t"
Nov 03 18:07:13.988589 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.988566    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b453729-cac0-4839-bdfa-0da04e59c1bc-workload-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t\" (UID: \"2b453729-cac0-4839-bdfa-0da04e59c1bc\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t"
Nov 03 18:07:13.988664 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:13.988600    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b453729-cac0-4839-bdfa-0da04e59c1bc-istio-envoy\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t\" (UID: \"2b453729-cac0-4839-bdfa-0da04e59c1bc\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t"
Nov 03 18:07:14.061491 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:14.061466    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n"]
Nov 03 18:07:14.068459 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:14.068423    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:07:14.068585 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:14.068491    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:07:14.068585 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:14.068518    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:07:14.090658 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:14.090626    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-workload-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82\" (UID: \"04fd7ef5-cf3c-4ed7-963d-e03294eaf327\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82"
Nov 03 18:07:14.090786 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:14.090668    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-workload-certs\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82\" (UID: \"04fd7ef5-cf3c-4ed7-963d-e03294eaf327\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82"
Nov 03 18:07:14.090786 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:14.090756    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-istio-envoy\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82\" (UID: \"04fd7ef5-cf3c-4ed7-963d-e03294eaf327\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82"
Nov 03 18:07:14.090924 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:14.090785    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b453729-cac0-4839-bdfa-0da04e59c1bc-istio-token\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t\" (UID: \"2b453729-cac0-4839-bdfa-0da04e59c1bc\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t"
Nov 03 18:07:14.090924 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:14.090808    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-istio-podinfo\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82\" (UID: \"04fd7ef5-cf3c-4ed7-963d-e03294eaf327\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82"
Nov 03 18:07:14.090924 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:14.090830    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-credential-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82\" (UID: \"04fd7ef5-cf3c-4ed7-963d-e03294eaf327\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82"
Nov 03 18:07:14.090924 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:14.090849    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b453729-cac0-4839-bdfa-0da04e59c1bc-istiod-ca-cert\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t\" (UID: \"2b453729-cac0-4839-bdfa-0da04e59c1bc\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t"
Nov 03 18:07:14.090924 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:14.090897    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-istio-token\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82\" (UID: \"04fd7ef5-cf3c-4ed7-963d-e03294eaf327\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82"
Nov 03 18:07:14.091174 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:14.090937    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b453729-cac0-4839-bdfa-0da04e59c1bc-istio-data\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t\" (UID: \"2b453729-cac0-4839-bdfa-0da04e59c1bc\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t"
Nov 03 18:07:14.091174 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:14.090971    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b453729-cac0-4839-bdfa-0da04e59c1bc-workload-certs\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t\" (UID: \"2b453729-cac0-4839-bdfa-0da04e59c1bc\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t"
Nov 03 18:07:14.091174 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:14.091003    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b453729-cac0-4839-bdfa-0da04e59c1bc-workload-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t\" (UID: \"2b453729-cac0-4839-bdfa-0da04e59c1bc\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t"
Nov 03 18:07:14.091174 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:14.091028    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b453729-cac0-4839-bdfa-0da04e59c1bc-istio-envoy\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t\" (UID: \"2b453729-cac0-4839-bdfa-0da04e59c1bc\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t"
Nov 03 18:07:14.091174 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:14.091074    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-workload-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82\" (UID: \"04fd7ef5-cf3c-4ed7-963d-e03294eaf327\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82"
Nov 03 18:07:14.091174 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:14.091130    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-workload-certs\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82\" (UID: \"04fd7ef5-cf3c-4ed7-963d-e03294eaf327\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82"
Nov 03 18:07:14.091467 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:14.091175    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klgts\" (UniqueName: \"kubernetes.io/projected/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-kube-api-access-klgts\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82\" (UID: \"04fd7ef5-cf3c-4ed7-963d-e03294eaf327\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82"
Nov 03 18:07:14.091467 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:14.091232    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b453729-cac0-4839-bdfa-0da04e59c1bc-credential-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t\" (UID: \"2b453729-cac0-4839-bdfa-0da04e59c1bc\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t"
Nov 03 18:07:14.091467 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:14.091279    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f55pm\" (UniqueName: \"kubernetes.io/projected/2b453729-cac0-4839-bdfa-0da04e59c1bc-kube-api-access-f55pm\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t\" (UID: \"2b453729-cac0-4839-bdfa-0da04e59c1bc\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t"
Nov 03 18:07:14.091467 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:14.091314    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-istio-data\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82\" (UID: \"04fd7ef5-cf3c-4ed7-963d-e03294eaf327\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82"
Nov 03 18:07:14.091467 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:14.091346    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-istiod-ca-cert\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82\" (UID: \"04fd7ef5-cf3c-4ed7-963d-e03294eaf327\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82"
Nov 03 18:07:14.091467 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:14.091373    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b453729-cac0-4839-bdfa-0da04e59c1bc-istio-podinfo\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t\" (UID: \"2b453729-cac0-4839-bdfa-0da04e59c1bc\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t"
Nov 03 18:07:14.091467 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:14.091386    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-credential-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82\" (UID: \"04fd7ef5-cf3c-4ed7-963d-e03294eaf327\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82"
Nov 03 18:07:14.091873 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:14.091754    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b453729-cac0-4839-bdfa-0da04e59c1bc-credential-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t\" (UID: \"2b453729-cac0-4839-bdfa-0da04e59c1bc\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t"
Nov 03 18:07:14.091873 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:14.091806    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b453729-cac0-4839-bdfa-0da04e59c1bc-workload-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t\" (UID: \"2b453729-cac0-4839-bdfa-0da04e59c1bc\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t"
Nov 03 18:07:14.091977 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:14.091912    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b453729-cac0-4839-bdfa-0da04e59c1bc-istiod-ca-cert\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t\" (UID: \"2b453729-cac0-4839-bdfa-0da04e59c1bc\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t"
Nov 03 18:07:14.092183 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:14.092161    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b453729-cac0-4839-bdfa-0da04e59c1bc-workload-certs\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t\" (UID: \"2b453729-cac0-4839-bdfa-0da04e59c1bc\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t"
Nov 03 18:07:14.092267 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:14.092187    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-istio-data\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82\" (UID: \"04fd7ef5-cf3c-4ed7-963d-e03294eaf327\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82"
Nov 03 18:07:14.092267 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:14.092252    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-istiod-ca-cert\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82\" (UID: \"04fd7ef5-cf3c-4ed7-963d-e03294eaf327\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82"
Nov 03 18:07:14.092417 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:14.092398    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b453729-cac0-4839-bdfa-0da04e59c1bc-istio-data\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t\" (UID: \"2b453729-cac0-4839-bdfa-0da04e59c1bc\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t"
Nov 03 18:07:14.095062 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:14.095039    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-istio-envoy\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82\" (UID: \"04fd7ef5-cf3c-4ed7-963d-e03294eaf327\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82"
Nov 03 18:07:14.095173 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:14.095118    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b453729-cac0-4839-bdfa-0da04e59c1bc-istio-podinfo\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t\" (UID: \"2b453729-cac0-4839-bdfa-0da04e59c1bc\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t"
Nov 03 18:07:14.095360 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:14.095329    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-istio-podinfo\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82\" (UID: \"04fd7ef5-cf3c-4ed7-963d-e03294eaf327\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82"
Nov 03 18:07:14.096176 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:14.096159    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b453729-cac0-4839-bdfa-0da04e59c1bc-istio-envoy\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t\" (UID: \"2b453729-cac0-4839-bdfa-0da04e59c1bc\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t"
Nov 03 18:07:14.152635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:14.152108    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-istio-token\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82\" (UID: \"04fd7ef5-cf3c-4ed7-963d-e03294eaf327\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82"
Nov 03 18:07:14.152823 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:14.152728    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klgts\" (UniqueName: \"kubernetes.io/projected/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-kube-api-access-klgts\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82\" (UID: \"04fd7ef5-cf3c-4ed7-963d-e03294eaf327\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82"
Nov 03 18:07:14.153036 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:14.152950    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f55pm\" (UniqueName: \"kubernetes.io/projected/2b453729-cac0-4839-bdfa-0da04e59c1bc-kube-api-access-f55pm\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t\" (UID: \"2b453729-cac0-4839-bdfa-0da04e59c1bc\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t"
Nov 03 18:07:14.157729 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:14.156565    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b453729-cac0-4839-bdfa-0da04e59c1bc-istio-token\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t\" (UID: \"2b453729-cac0-4839-bdfa-0da04e59c1bc\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t"
Nov 03 18:07:14.171871 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:14.171846    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82"
Nov 03 18:07:14.188270 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:14.188240    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t"
Nov 03 18:07:14.512482 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:14.510337    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn"
Nov 03 18:07:14.516243 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:14.514994    5315 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 03 18:07:14.516243 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:14.515056    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn" podUID="dafbc7c7-538e-4bf6-ab30-970870f21854" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 03 18:07:15.250375 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:15.249932    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l" podUID="114dcf54-e4eb-4294-a77b-604fe6d57d48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:15.296776 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:15.296733    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7" podUID="69760dba-faf9-4c08-895c-ad70fe95fd49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:15.373828 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:15.316016    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s" podUID="155e4b39-1877-4bd9-8541-5d6bd3aa0a8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:15.373828 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:15.328722    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q" podUID="c974b0f1-e00b-4e4a-b193-df769068983d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:15.373828 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:15.328763    5315 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q" podUID="c974b0f1-e00b-4e4a-b193-df769068983d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded"
Nov 03 18:07:15.373828 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:15.337014    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7" podUID="09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:15.393649 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:15.380802    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr" podUID="08159a02-fc2e-4138-824d-be5ba3e141d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:15.409676 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:15.408277    5315 generic.go:356] "Generic (PLEG): container finished" podID="8f5bff5e-4b15-4cde-a3b0-71c9ae666122" containerID="61bcac3edd0b11569024e70049f166240dc379a0b07b8d6febfe7be692b1e5f7" exitCode=0
Nov 03 18:07:15.409676 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:15.408423    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr" event={"ID":"8f5bff5e-4b15-4cde-a3b0-71c9ae666122","Type":"ContainerDied","Data":"61bcac3edd0b11569024e70049f166240dc379a0b07b8d6febfe7be692b1e5f7"}
Nov 03 18:07:15.441659 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:15.430208    5315 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 03 18:07:15.441659 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:15.430271    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn" podUID="dafbc7c7-538e-4bf6-ab30-970870f21854" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 03 18:07:15.471938 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:15.462482    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c" podUID="8ac59539-8ac1-466e-8d1b-b2be4c42c708" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Nov 03 18:07:15.798236 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:15.635191    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82"]
Nov 03 18:07:15.798236 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:15.728182    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv" podUID="f6d706cb-3aa5-419c-8f6c-29347bb788b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 18:07:15.798236 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:15.759188    5315 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr" podUID="df8e0b71-e876-4cea-a7ae-3532c887c660" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:16.225595 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:16.225489    5315 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7" podUID="69760dba-faf9-4c08-895c-ad70fe95fd49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:16.226218 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:16.226184    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt" podUID="7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Nov 03 18:07:16.339239 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:16.339184    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr" podUID="df8e0b71-e876-4cea-a7ae-3532c887c660" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:16.348098 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:16.347268    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7" podUID="69760dba-faf9-4c08-895c-ad70fe95fd49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:16.367958 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:16.367898    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:07:16.368231 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:16.368206    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:07:16.368452 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:16.368430    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:07:16.368977 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:16.368941    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7" podUID="09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:16.370050 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:16.370021    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s" podUID="155e4b39-1877-4bd9-8541-5d6bd3aa0a8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Nov 03 18:07:16.370463 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:16.370433    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q" podUID="c974b0f1-e00b-4e4a-b193-df769068983d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 18:07:16.390631 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:16.383519    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-248f8ee3-574c4c5b5-rqjv5"]
Nov 03 18:07:16.410660 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:16.404210    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l" podUID="114dcf54-e4eb-4294-a77b-604fe6d57d48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:16.536674 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:16.535245    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr" podUID="08159a02-fc2e-4138-824d-be5ba3e141d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded"
Nov 03 18:07:16.660770 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:16.660714    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-r2jgr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 18:07:16.788899 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:16.746404    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr" podUID="8f5bff5e-4b15-4cde-a3b0-71c9ae666122" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 18:07:16.788899 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:16.661881    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c" podUID="8ac59539-8ac1-466e-8d1b-b2be4c42c708" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:16.813637 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:16.802600    5315 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 03 18:07:16.813637 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:16.802694    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn" podUID="dafbc7c7-538e-4bf6-ab30-970870f21854" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 03 18:07:16.845780 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:16.845734    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7" podUID="33725a51-6459-4f15-a866-e9dee41a9732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 03 18:07:16.993642 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:16.907515    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 18:07:17.008635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:17.004720    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n" podUID="1e4ca152-3632-485f-94b2-4f430cbdfe4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 18:07:17.047318 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:16.980855    5315 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l" podUID="114dcf54-e4eb-4294-a77b-604fe6d57d48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:17.093635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:16.939265    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv" podUID="f6d706cb-3aa5-419c-8f6c-29347bb788b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 18:07:17.226803 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:17.226747    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt" podUID="7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:17.268698 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:17.268637    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:07:17.318710 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:17.318567    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:07:17.319015 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:17.318986    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:07:17.353297 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:17.330860    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-248f8ee3-574c4c5b5-rqjv5"]
Nov 03 18:07:17.353297 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:17.330921    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n"
Nov 03 18:07:17.353297 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:17.330956    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-248f8ee3-576cfc4795-5q9nf"]
Nov 03 18:07:17.353297 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:17.336840    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-248f8ee3-574c4c5b5-rqjv5"
Nov 03 18:07:17.355336 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:17.355290    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7" podUID="69760dba-faf9-4c08-895c-ad70fe95fd49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:17.356437 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:17.355627    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr" podUID="df8e0b71-e876-4cea-a7ae-3532c887c660" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 18:07:17.397344 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:17.377999    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s" podUID="155e4b39-1877-4bd9-8541-5d6bd3aa0a8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:17.397344 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:17.388445    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q" podUID="c974b0f1-e00b-4e4a-b193-df769068983d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:17.397344 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:17.392734    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7" podUID="09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Nov 03 18:07:17.439596 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:17.406521    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-248f8ee3\""
Nov 03 18:07:17.439596 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:17.408116    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l" podUID="114dcf54-e4eb-4294-a77b-604fe6d57d48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:17.552835 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:17.552724    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr" podUID="08159a02-fc2e-4138-824d-be5ba3e141d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:17.554368 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:17.554290    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ced2f03-d1ad-4a25-aa71-d6ece3178be9-plugins-config-volume\") pod \"payload-processing-e2e-ait-248f8ee3-574c4c5b5-rqjv5\" (UID: \"1ced2f03-d1ad-4a25-aa71-d6ece3178be9\") " pod="openshift-ingress/payload-processing-e2e-ait-248f8ee3-574c4c5b5-rqjv5"
Nov 03 18:07:17.554511 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:17.554411    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sff2\" (UniqueName: \"kubernetes.io/projected/1ced2f03-d1ad-4a25-aa71-d6ece3178be9-kube-api-access-7sff2\") pod \"payload-processing-e2e-ait-248f8ee3-574c4c5b5-rqjv5\" (UID: \"1ced2f03-d1ad-4a25-aa71-d6ece3178be9\") " pod="openshift-ingress/payload-processing-e2e-ait-248f8ee3-574c4c5b5-rqjv5"
Nov 03 18:07:17.664052 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:17.663950    5315 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 03 18:07:17.664317 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:17.664283    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn" podUID="dafbc7c7-538e-4bf6-ab30-970870f21854" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 03 18:07:17.715178 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:17.715128    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n" event={"ID":"1e4ca152-3632-485f-94b2-4f430cbdfe4c","Type":"ContainerStarted","Data":"fc9e4e62d13aca2b8af6f9ad6220a16317402c53efa4273edf35cdb15050d371"}
Nov 03 18:07:17.715442 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:17.715421    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-248f8ee3-576cfc4795-5q9nf"]
Nov 03 18:07:17.746509 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:17.731236    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ced2f03-d1ad-4a25-aa71-d6ece3178be9-plugins-config-volume\") pod \"payload-processing-e2e-ait-248f8ee3-574c4c5b5-rqjv5\" (UID: \"1ced2f03-d1ad-4a25-aa71-d6ece3178be9\") " pod="openshift-ingress/payload-processing-e2e-ait-248f8ee3-574c4c5b5-rqjv5"
Nov 03 18:07:17.746509 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:17.731341    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sff2\" (UniqueName: \"kubernetes.io/projected/1ced2f03-d1ad-4a25-aa71-d6ece3178be9-kube-api-access-7sff2\") pod \"payload-processing-e2e-ait-248f8ee3-574c4c5b5-rqjv5\" (UID: \"1ced2f03-d1ad-4a25-aa71-d6ece3178be9\") " pod="openshift-ingress/payload-processing-e2e-ait-248f8ee3-574c4c5b5-rqjv5"
Nov 03 18:07:17.791895 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:17.787933    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 18:07:17.791895 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:17.787990    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n" podUID="1e4ca152-3632-485f-94b2-4f430cbdfe4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 18:07:17.791895 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:17.789408    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c" podUID="8ac59539-8ac1-466e-8d1b-b2be4c42c708" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:17.811975 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:17.793747    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-248f8ee3-576cfc4795-5q9nf"
Nov 03 18:07:17.898640 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:17.872918    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7j6q\" (UniqueName: \"kubernetes.io/projected/05885de3-0afb-4ae7-a7d2-7c3b64e0bba1-kube-api-access-j7j6q\") pod \"payload-pre-processing-e2e-ait-248f8ee3-576cfc4795-5q9nf\" (UID: \"05885de3-0afb-4ae7-a7d2-7c3b64e0bba1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-248f8ee3-576cfc4795-5q9nf"
Nov 03 18:07:17.898640 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:17.873054    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05885de3-0afb-4ae7-a7d2-7c3b64e0bba1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-248f8ee3-576cfc4795-5q9nf\" (UID: \"05885de3-0afb-4ae7-a7d2-7c3b64e0bba1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-248f8ee3-576cfc4795-5q9nf"
Nov 03 18:07:17.940696 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:17.900782    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ced2f03-d1ad-4a25-aa71-d6ece3178be9-plugins-config-volume\") pod \"payload-processing-e2e-ait-248f8ee3-574c4c5b5-rqjv5\" (UID: \"1ced2f03-d1ad-4a25-aa71-d6ece3178be9\") " pod="openshift-ingress/payload-processing-e2e-ait-248f8ee3-574c4c5b5-rqjv5"
Nov 03 18:07:17.953636 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:17.951563    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7" podUID="33725a51-6459-4f15-a866-e9dee41a9732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 03 18:07:18.176552 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:18.052902    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7j6q\" (UniqueName: \"kubernetes.io/projected/05885de3-0afb-4ae7-a7d2-7c3b64e0bba1-kube-api-access-j7j6q\") pod \"payload-pre-processing-e2e-ait-248f8ee3-576cfc4795-5q9nf\" (UID: \"05885de3-0afb-4ae7-a7d2-7c3b64e0bba1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-248f8ee3-576cfc4795-5q9nf"
Nov 03 18:07:18.176552 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:18.089601    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05885de3-0afb-4ae7-a7d2-7c3b64e0bba1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-248f8ee3-576cfc4795-5q9nf\" (UID: \"05885de3-0afb-4ae7-a7d2-7c3b64e0bba1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-248f8ee3-576cfc4795-5q9nf"
Nov 03 18:07:18.176552 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:18.145742    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv" podUID="f6d706cb-3aa5-419c-8f6c-29347bb788b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:18.179413 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:18.179375    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05885de3-0afb-4ae7-a7d2-7c3b64e0bba1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-248f8ee3-576cfc4795-5q9nf\" (UID: \"05885de3-0afb-4ae7-a7d2-7c3b64e0bba1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-248f8ee3-576cfc4795-5q9nf"
Nov 03 18:07:18.213138 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:18.181333    5315 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv" podUID="f6d706cb-3aa5-419c-8f6c-29347bb788b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": context deadline exceeded"
Nov 03 18:07:18.213360 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:18.190493    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sff2\" (UniqueName: \"kubernetes.io/projected/1ced2f03-d1ad-4a25-aa71-d6ece3178be9-kube-api-access-7sff2\") pod \"payload-processing-e2e-ait-248f8ee3-574c4c5b5-rqjv5\" (UID: \"1ced2f03-d1ad-4a25-aa71-d6ece3178be9\") " pod="openshift-ingress/payload-processing-e2e-ait-248f8ee3-574c4c5b5-rqjv5"
Nov 03 18:07:18.213478 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:18.206697    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-248f8ee3-dockercfg-bqmm4\""
Nov 03 18:07:18.256907 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:18.227834    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt" podUID="7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:18.256907 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:18.243106    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-248f8ee3-574c4c5b5-rqjv5"
Nov 03 18:07:18.423637 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:18.350395    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7j6q\" (UniqueName: \"kubernetes.io/projected/05885de3-0afb-4ae7-a7d2-7c3b64e0bba1-kube-api-access-j7j6q\") pod \"payload-pre-processing-e2e-ait-248f8ee3-576cfc4795-5q9nf\" (UID: \"05885de3-0afb-4ae7-a7d2-7c3b64e0bba1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-248f8ee3-576cfc4795-5q9nf"
Nov 03 18:07:18.423637 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:18.362815    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr" podUID="df8e0b71-e876-4cea-a7ae-3532c887c660" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:18.462340 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:18.457896    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7" podUID="69760dba-faf9-4c08-895c-ad70fe95fd49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/ready\": context deadline exceeded"
Nov 03 18:07:18.483872 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:18.480862    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n" event={"ID":"1e4ca152-3632-485f-94b2-4f430cbdfe4c","Type":"ContainerStarted","Data":"7b1ea837003b2acfec8f1382c79898423ca89d7ec70da97b28504667dbda28ec"}
Nov 03 18:07:18.483872 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:18.480986    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7"
Nov 03 18:07:18.483872 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:18.481028    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7"
Nov 03 18:07:18.483872 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:18.481063    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s"
Nov 03 18:07:18.483872 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:18.481096    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q"
Nov 03 18:07:18.483872 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:18.481154    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l"
Nov 03 18:07:18.497147 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:18.481186    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr"
Nov 03 18:07:18.576674 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:18.562696    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c"
Nov 03 18:07:18.576674 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:18.562849    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv"
Nov 03 18:07:18.576674 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:18.562892    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt"
Nov 03 18:07:18.576674 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:18.562929    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr"
Nov 03 18:07:18.576674 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:18.562963    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7"
Nov 03 18:07:18.576674 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:18.563272    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr" podUID="08159a02-fc2e-4138-824d-be5ba3e141d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:18.581792 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:18.581748    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s" podUID="155e4b39-1877-4bd9-8541-5d6bd3aa0a8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Nov 03 18:07:18.582787 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:18.582756    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q" podUID="c974b0f1-e00b-4e4a-b193-df769068983d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 18:07:18.584130 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:18.584105    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7" podUID="09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Nov 03 18:07:18.584805 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:18.584781    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l" podUID="114dcf54-e4eb-4294-a77b-604fe6d57d48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:18.654770 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:18.608517    5315 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 03 18:07:18.654770 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:18.608581    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn" podUID="dafbc7c7-538e-4bf6-ab30-970870f21854" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 03 18:07:18.654770 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:18.617790    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-248f8ee3-576cfc4795-5q9nf"
Nov 03 18:07:18.654770 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:18.618149    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n" podStartSLOduration=5.61813071 podStartE2EDuration="5.61813071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:07:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:07:16.549304557 +0000 UTC m=+1275.565502631" watchObservedRunningTime="2026-11-03 18:07:18.61813071 +0000 UTC m=+1277.634328780"
Nov 03 18:07:18.654770 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:18.631287    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t"]
Nov 03 18:07:18.691655 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:18.691592    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 18:07:18.705189 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:18.695757    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n" podUID="1e4ca152-3632-485f-94b2-4f430cbdfe4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 18:07:18.842536 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:18.840177    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c" podUID="8ac59539-8ac1-466e-8d1b-b2be4c42c708" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Nov 03 18:07:19.019184 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:19.019132    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7" podUID="33725a51-6459-4f15-a866-e9dee41a9732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:19.072304 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:19.072263    5315 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7" podUID="09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": context deadline exceeded"
Nov 03 18:07:19.151636 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:19.151083    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82" event={"ID":"04fd7ef5-cf3c-4ed7-963d-e03294eaf327","Type":"ContainerStarted","Data":"f3db370de73d2f1a513cd971b77a0695d7e9fbfc7cbf62c5069e66013443e82e"}
Nov 03 18:07:19.153368 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:19.150976    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv" podUID="f6d706cb-3aa5-419c-8f6c-29347bb788b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:19.247254 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:19.247208    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt" podUID="7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Nov 03 18:07:19.387634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:19.384957    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr" podUID="df8e0b71-e876-4cea-a7ae-3532c887c660" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 18:07:19.447514 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:19.446099    5315 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 03 18:07:19.447514 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:19.446163    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn" podUID="dafbc7c7-538e-4bf6-ab30-970870f21854" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 03 18:07:19.475926 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:19.475886    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7" podUID="69760dba-faf9-4c08-895c-ad70fe95fd49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:19.500097 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:19.495179    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr"
Nov 03 18:07:19.568762 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:19.568719    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr" podUID="08159a02-fc2e-4138-824d-be5ba3e141d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:19.584647 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:19.583724    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q" podUID="c974b0f1-e00b-4e4a-b193-df769068983d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:19.584647 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:19.584144    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s" podUID="155e4b39-1877-4bd9-8541-5d6bd3aa0a8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Nov 03 18:07:19.584647 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:19.584460    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7" podUID="09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:19.585093 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:19.585059    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l" podUID="114dcf54-e4eb-4294-a77b-604fe6d57d48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:19.647284 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:19.647145    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-workload-socket\") pod \"8f5bff5e-4b15-4cde-a3b0-71c9ae666122\" (UID: \"8f5bff5e-4b15-4cde-a3b0-71c9ae666122\") "
Nov 03 18:07:19.649716 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:19.648124    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-istio-data\") pod \"8f5bff5e-4b15-4cde-a3b0-71c9ae666122\" (UID: \"8f5bff5e-4b15-4cde-a3b0-71c9ae666122\") "
Nov 03 18:07:19.649716 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:19.648186    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-istio-podinfo\") pod \"8f5bff5e-4b15-4cde-a3b0-71c9ae666122\" (UID: \"8f5bff5e-4b15-4cde-a3b0-71c9ae666122\") "
Nov 03 18:07:19.649716 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:19.648269    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-istio-token\" (UniqueName: \"kubernetes.io/projected/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-istio-token\") pod \"8f5bff5e-4b15-4cde-a3b0-71c9ae666122\" (UID: \"8f5bff5e-4b15-4cde-a3b0-71c9ae666122\") "
Nov 03 18:07:19.649716 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:19.648360    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-kube-api-access-xbl24\" (UniqueName: \"kubernetes.io/projected/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-kube-api-access-xbl24\") pod \"8f5bff5e-4b15-4cde-a3b0-71c9ae666122\" (UID: \"8f5bff5e-4b15-4cde-a3b0-71c9ae666122\") "
Nov 03 18:07:19.649716 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:19.648434    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-credential-socket\") pod \"8f5bff5e-4b15-4cde-a3b0-71c9ae666122\" (UID: \"8f5bff5e-4b15-4cde-a3b0-71c9ae666122\") "
Nov 03 18:07:19.649716 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:19.648581    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-workload-certs\") pod \"8f5bff5e-4b15-4cde-a3b0-71c9ae666122\" (UID: \"8f5bff5e-4b15-4cde-a3b0-71c9ae666122\") "
Nov 03 18:07:19.649716 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:19.648706    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-istio-envoy\") pod \"8f5bff5e-4b15-4cde-a3b0-71c9ae666122\" (UID: \"8f5bff5e-4b15-4cde-a3b0-71c9ae666122\") "
Nov 03 18:07:19.649716 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:19.648754    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-istiod-ca-cert\") pod \"8f5bff5e-4b15-4cde-a3b0-71c9ae666122\" (UID: \"8f5bff5e-4b15-4cde-a3b0-71c9ae666122\") "
Nov 03 18:07:19.650735 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:19.650457    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-istiod-ca-cert" pod "8f5bff5e-4b15-4cde-a3b0-71c9ae666122" (UID: "8f5bff5e-4b15-4cde-a3b0-71c9ae666122"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:07:19.650735 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:19.650689    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-workload-socket" pod "8f5bff5e-4b15-4cde-a3b0-71c9ae666122" (UID: "8f5bff5e-4b15-4cde-a3b0-71c9ae666122"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:07:19.651630 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:19.651133    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-credential-socket" pod "8f5bff5e-4b15-4cde-a3b0-71c9ae666122" (UID: "8f5bff5e-4b15-4cde-a3b0-71c9ae666122"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:07:19.651630 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:19.651332    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-workload-certs" pod "8f5bff5e-4b15-4cde-a3b0-71c9ae666122" (UID: "8f5bff5e-4b15-4cde-a3b0-71c9ae666122"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:07:19.651932 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:19.651911    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-istio-data" pod "8f5bff5e-4b15-4cde-a3b0-71c9ae666122" (UID: "8f5bff5e-4b15-4cde-a3b0-71c9ae666122"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:07:19.747315 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:19.747279    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 18:07:19.747533 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:19.747512    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n" podUID="1e4ca152-3632-485f-94b2-4f430cbdfe4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 18:07:19.749792 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:19.749765    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr"
Nov 03 18:07:19.750058 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:19.750034    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t" event={"ID":"2b453729-cac0-4839-bdfa-0da04e59c1bc","Type":"ContainerStarted","Data":"f6c672ba463ccdc778d718267f0191dd7d5c2a52cf3a40c2947d7d3732fd3c3b"}
Nov 03 18:07:19.769645 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:19.758411    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:07:19.769986 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:19.769909    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:07:19.770174 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:19.770156    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:07:19.770328 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:19.770313    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:07:19.770448 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:19.770434    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:07:19.887114 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:19.868757    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c" podUID="8ac59539-8ac1-466e-8d1b-b2be4c42c708" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Nov 03 18:07:19.963685 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:19.936797    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-istio-podinfo" pod "8f5bff5e-4b15-4cde-a3b0-71c9ae666122" (UID: "8f5bff5e-4b15-4cde-a3b0-71c9ae666122"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:07:19.965085 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:19.965043    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-kube-api-access-xbl24" pod "8f5bff5e-4b15-4cde-a3b0-71c9ae666122" (UID: "8f5bff5e-4b15-4cde-a3b0-71c9ae666122"). InnerVolumeSpecName "kube-api-access-xbl24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:07:20.035988 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:20.035933    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7" podUID="33725a51-6459-4f15-a866-e9dee41a9732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 03 18:07:20.057937 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:20.038847    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:07:20.058211 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:20.058189    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbl24\" (UniqueName: \"kubernetes.io/projected/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-kube-api-access-xbl24\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:07:20.121830 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:20.115953    5315 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt" podUID="7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:20.288708 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:20.265317    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-istio-token" pod "8f5bff5e-4b15-4cde-a3b0-71c9ae666122" (UID: "8f5bff5e-4b15-4cde-a3b0-71c9ae666122"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:07:20.422392 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:20.314866    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-istio-envoy" pod "8f5bff5e-4b15-4cde-a3b0-71c9ae666122" (UID: "8f5bff5e-4b15-4cde-a3b0-71c9ae666122"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:07:20.422595 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:20.418624    5315 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 03 18:07:20.422780 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:20.422749    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn" podUID="dafbc7c7-538e-4bf6-ab30-970870f21854" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 03 18:07:20.422904 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:20.311938    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt" podUID="7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Nov 03 18:07:20.423298 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:20.314704    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv" podUID="f6d706cb-3aa5-419c-8f6c-29347bb788b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:20.497632 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:20.479323    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:07:20.497632 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:20.479364    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f5bff5e-4b15-4cde-a3b0-71c9ae666122-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:07:20.497632 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:20.481275    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7" podUID="69760dba-faf9-4c08-895c-ad70fe95fd49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:20.608743 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:20.572651    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr" podUID="08159a02-fc2e-4138-824d-be5ba3e141d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:20.620639 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:20.612380    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7" podUID="09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:20.636682 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:20.622989    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q" podUID="c974b0f1-e00b-4e4a-b193-df769068983d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 18:07:20.636682 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:20.623682    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s" podUID="155e4b39-1877-4bd9-8541-5d6bd3aa0a8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Nov 03 18:07:20.673670 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:20.648374    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l" podUID="114dcf54-e4eb-4294-a77b-604fe6d57d48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:20.673670 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:20.671166    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr" podUID="df8e0b71-e876-4cea-a7ae-3532c887c660" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:20.713637 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:20.677392    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 18:07:20.964321 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:20.777937    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n" podUID="1e4ca152-3632-485f-94b2-4f430cbdfe4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 18:07:21.006680 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:20.965103    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c" podUID="8ac59539-8ac1-466e-8d1b-b2be4c42c708" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:21.085656 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:21.072094    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7" podUID="33725a51-6459-4f15-a866-e9dee41a9732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:21.085656 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:21.078967    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82" event={"ID":"04fd7ef5-cf3c-4ed7-963d-e03294eaf327","Type":"ContainerStarted","Data":"e56a7f73492bd3b33294e91ede46de4ea9ee4b02e93ea8575c5f9dbc2f12167d"}
Nov 03 18:07:21.085656 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:21.084770    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t" event={"ID":"2b453729-cac0-4839-bdfa-0da04e59c1bc","Type":"ContainerStarted","Data":"0feeb0a45b8262c496c57b031f48c2254a1e7ed1a311b3614f093f5363fa1829"}
Nov 03 18:07:21.217681 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:21.184693    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82"
Nov 03 18:07:21.232462 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:21.232415    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Nov 03 18:07:21.232747 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:21.232707    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t" podUID="2b453729-cac0-4839-bdfa-0da04e59c1bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Nov 03 18:07:21.233035 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:21.233015    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 03 18:07:21.233257 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:21.233238    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82" podUID="04fd7ef5-cf3c-4ed7-963d-e03294eaf327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 03 18:07:21.506370 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:21.481789    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7" podUID="69760dba-faf9-4c08-895c-ad70fe95fd49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/ready\": context deadline exceeded"
Nov 03 18:07:21.528012 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:21.527962    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t"
Nov 03 18:07:21.621787 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:21.576773    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr" podUID="08159a02-fc2e-4138-824d-be5ba3e141d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:21.657468 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:21.624637    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7" podUID="09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:21.739674 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:21.713864    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l" podUID="114dcf54-e4eb-4294-a77b-604fe6d57d48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Nov 03 18:07:21.889633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:21.625571    5315 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 03 18:07:21.889633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:21.889254    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn" podUID="dafbc7c7-538e-4bf6-ab30-970870f21854" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 03 18:07:21.889633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:21.855268    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s" podUID="155e4b39-1877-4bd9-8541-5d6bd3aa0a8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Nov 03 18:07:21.895486 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:21.855231    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q" podUID="c974b0f1-e00b-4e4a-b193-df769068983d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 18:07:21.910634 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:07:21.702102    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ced2f03_d1ad_4a25_aa71_d6ece3178be9.slice/crio-84b91f01b9a96437f84a8d6b0dbfb015c4dd8d792bcdcaae0b517c4bf10c5b50 WatchSource:0}: Error finding container 84b91f01b9a96437f84a8d6b0dbfb015c4dd8d792bcdcaae0b517c4bf10c5b50: Status 404 returned error can't find the container with id 84b91f01b9a96437f84a8d6b0dbfb015c4dd8d792bcdcaae0b517c4bf10c5b50
Nov 03 18:07:21.910634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:21.855344    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 18:07:21.910634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:21.907746    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n" podUID="1e4ca152-3632-485f-94b2-4f430cbdfe4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 18:07:21.910634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:21.730162    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr" podUID="df8e0b71-e876-4cea-a7ae-3532c887c660" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 18:07:21.917902 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:21.913577    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt" podUID="7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Nov 03 18:07:21.917902 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:21.913905    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv" podUID="f6d706cb-3aa5-419c-8f6c-29347bb788b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 18:07:21.982635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:21.981716    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr"
Nov 03 18:07:22.029475 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:22.012144    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c" podUID="8ac59539-8ac1-466e-8d1b-b2be4c42c708" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:22.199895 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:22.199848    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7" podUID="33725a51-6459-4f15-a866-e9dee41a9732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 03 18:07:22.207662 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:22.204030    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx" podUID="626f43f8-df88-48d5-9f11-ca75310a0fc2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.22:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:22.231640 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:22.220272    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Nov 03 18:07:22.231640 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:22.220344    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t" podUID="2b453729-cac0-4839-bdfa-0da04e59c1bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Nov 03 18:07:22.310766 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:22.310674    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-d6rcp" podUID="ee0c4b5d-37dd-482b-b5e9-4256a332a556" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/ready\": context deadline exceeded"
Nov 03 18:07:22.349091 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:22.346372    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 03 18:07:22.349091 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:22.346453    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82" podUID="04fd7ef5-cf3c-4ed7-963d-e03294eaf327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 03 18:07:22.430711 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:22.430670    5315 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 03 18:07:22.430996 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:22.430961    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn" podUID="dafbc7c7-538e-4bf6-ab30-970870f21854" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 03 18:07:22.514787 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:22.514723    5315 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c" podUID="8ac59539-8ac1-466e-8d1b-b2be4c42c708" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:22.523638 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:22.516663    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7" podUID="69760dba-faf9-4c08-895c-ad70fe95fd49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/ready\": context deadline exceeded"
Nov 03 18:07:22.570836 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:22.570790    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr" event={"ID":"8f5bff5e-4b15-4cde-a3b0-71c9ae666122","Type":"ContainerDied","Data":"e204e645d8cbbe6f47ef145907e613050a99e12e29b66f5427019eabe09d8169"}
Nov 03 18:07:22.592750 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:22.592705    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr"
Nov 03 18:07:22.594795 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:22.592978    5315 scope.go:122] "RemoveContainer" containerID="61bcac3edd0b11569024e70049f166240dc379a0b07b8d6febfe7be692b1e5f7"
Nov 03 18:07:22.667481 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:22.667446    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr"
Nov 03 18:07:22.679969 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:22.679931    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 18:07:22.680312 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:22.680278    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n" podUID="1e4ca152-3632-485f-94b2-4f430cbdfe4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 18:07:22.754306 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:22.694191    5315 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s" podUID="155e4b39-1877-4bd9-8541-5d6bd3aa0a8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:22.794933 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:22.752741    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr" podUID="08159a02-fc2e-4138-824d-be5ba3e141d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded"
Nov 03 18:07:22.796741 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:22.752790    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7" podUID="09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Nov 03 18:07:22.924825 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:22.924711    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv" podUID="f6d706cb-3aa5-419c-8f6c-29347bb788b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:22.925728 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:22.925693    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt" podUID="7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Nov 03 18:07:22.965599 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:22.965560    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7"
Nov 03 18:07:23.012381 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:22.989568    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q" podUID="c974b0f1-e00b-4e4a-b193-df769068983d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:23.012381 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:22.990976    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s" podUID="155e4b39-1877-4bd9-8541-5d6bd3aa0a8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:23.019854 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:23.019792    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l" podUID="114dcf54-e4eb-4294-a77b-604fe6d57d48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:23.197858 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:23.072122    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c" podUID="8ac59539-8ac1-466e-8d1b-b2be4c42c708" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Nov 03 18:07:23.197858 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:23.176326    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 03 18:07:23.197858 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:23.180518    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82" podUID="04fd7ef5-cf3c-4ed7-963d-e03294eaf327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 03 18:07:23.215653 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:23.206503    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Nov 03 18:07:23.215653 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:23.206558    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t" podUID="2b453729-cac0-4839-bdfa-0da04e59c1bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Nov 03 18:07:23.374636 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:23.360697    5315 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7" podUID="33725a51-6459-4f15-a866-e9dee41a9732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:23.467872 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:23.453694    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-248f8ee3-574c4c5b5-rqjv5" event={"ID":"1ced2f03-d1ad-4a25-aa71-d6ece3178be9","Type":"ContainerStarted","Data":"84b91f01b9a96437f84a8d6b0dbfb015c4dd8d792bcdcaae0b517c4bf10c5b50"}
Nov 03 18:07:23.467872 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:23.454096    5315 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 03 18:07:23.467872 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:23.454146    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn" podUID="dafbc7c7-538e-4bf6-ab30-970870f21854" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 03 18:07:23.540962 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:23.535080    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7" podUID="69760dba-faf9-4c08-895c-ad70fe95fd49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:23.703165 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:23.703126    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 18:07:23.703448 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:23.703412    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n" podUID="1e4ca152-3632-485f-94b2-4f430cbdfe4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 18:07:23.838861 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:23.838809    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr" podUID="08159a02-fc2e-4138-824d-be5ba3e141d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded"
Nov 03 18:07:23.850396 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:23.850342    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7" podUID="09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Nov 03 18:07:23.897858 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:23.897816    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr" podUID="df8e0b71-e876-4cea-a7ae-3532c887c660" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:23.931854 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:23.926756    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv" podUID="f6d706cb-3aa5-419c-8f6c-29347bb788b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:23.931854 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:23.930888    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt" podUID="7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Nov 03 18:07:23.997637 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:23.994305    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s" podUID="155e4b39-1877-4bd9-8541-5d6bd3aa0a8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:23.997637 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:23.994677    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q" podUID="c974b0f1-e00b-4e4a-b193-df769068983d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 18:07:24.038925 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:24.038881    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7" podUID="33725a51-6459-4f15-a866-e9dee41a9732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:24.039217 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:24.039194    5315 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr" podUID="08159a02-fc2e-4138-824d-be5ba3e141d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": context deadline exceeded"
Nov 03 18:07:24.117651 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:24.101112    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-248f8ee3-574c4c5b5-rqjv5"]
Nov 03 18:07:24.180187 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:24.180144    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c" podUID="8ac59539-8ac1-466e-8d1b-b2be4c42c708" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:24.180756 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:24.180728    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l" podUID="114dcf54-e4eb-4294-a77b-604fe6d57d48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Nov 03 18:07:24.257665 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:24.252515    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 03 18:07:24.257665 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:24.252581    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82" podUID="04fd7ef5-cf3c-4ed7-963d-e03294eaf327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 03 18:07:24.257665 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:24.252697    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Nov 03 18:07:24.257665 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:24.252723    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t" podUID="2b453729-cac0-4839-bdfa-0da04e59c1bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Nov 03 18:07:24.432831 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:24.432656    5315 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 03 18:07:24.433056 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:24.432991    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn" podUID="dafbc7c7-538e-4bf6-ab30-970870f21854" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 03 18:07:24.505356 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:24.505289    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t" podStartSLOduration=11.505265024 podStartE2EDuration="11.505265024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:07:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:07:22.21567472 +0000 UTC m=+1281.231872792" watchObservedRunningTime="2026-11-03 18:07:24.505265024 +0000 UTC m=+1283.521463095"
Nov 03 18:07:24.518599 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:24.518530    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82" podStartSLOduration=11.518509775 podStartE2EDuration="11.518509775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:07:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:07:22.663843207 +0000 UTC m=+1281.680041277" watchObservedRunningTime="2026-11-03 18:07:24.518509775 +0000 UTC m=+1283.534707848"
Nov 03 18:07:24.541259 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:24.541214    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7" podUID="69760dba-faf9-4c08-895c-ad70fe95fd49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:24.676737 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:24.666286    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 18:07:24.676737 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:24.666341    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n" podUID="1e4ca152-3632-485f-94b2-4f430cbdfe4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 18:07:24.781988 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:24.781952    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-248f8ee3-576cfc4795-5q9nf"]
Nov 03 18:07:24.796850 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:24.796809    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-248f8ee3-574c4c5b5-rqjv5" event={"ID":"1ced2f03-d1ad-4a25-aa71-d6ece3178be9","Type":"ContainerStarted","Data":"db8941ae2d8da604976cbe28d6e9658d8875a2106d9612ed4356380a82ece702"}
Nov 03 18:07:24.798773 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:24.798748    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-248f8ee3-574c4c5b5-rqjv5"
Nov 03 18:07:24.842633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:24.839819    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr" podUID="08159a02-fc2e-4138-824d-be5ba3e141d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:24.864502 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:07:24.864459    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:07:24.866806 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:24.866756    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7" podUID="09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:24.902382 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:24.902341    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-248f8ee3-576cfc4795-5q9nf" event={"ID":"05885de3-0afb-4ae7-a7d2-7c3b64e0bba1","Type":"ContainerStarted","Data":"dceef1823463c0aee0af310cd70c06e9d1193080b8e7bbcc0e582979e8325762"}
Nov 03 18:07:25.009684 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:24.984285    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr" podUID="df8e0b71-e876-4cea-a7ae-3532c887c660" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:25.009684 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:24.984721    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt" podUID="7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:25.009684 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:24.984849    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv" podUID="f6d706cb-3aa5-419c-8f6c-29347bb788b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 18:07:25.012330 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:25.011797    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr"
Nov 03 18:07:25.021674 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:25.013418    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q" podUID="c974b0f1-e00b-4e4a-b193-df769068983d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:25.021674 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:25.013659    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s" podUID="155e4b39-1877-4bd9-8541-5d6bd3aa0a8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Nov 03 18:07:25.057698 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:25.042631    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7" podUID="33725a51-6459-4f15-a866-e9dee41a9732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:25.095310 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:25.058045    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7"
Nov 03 18:07:25.207633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:25.185775    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l" podUID="114dcf54-e4eb-4294-a77b-604fe6d57d48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Nov 03 18:07:25.207633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:25.195975    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c" podUID="8ac59539-8ac1-466e-8d1b-b2be4c42c708" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Nov 03 18:07:25.310810 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:25.310744    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Nov 03 18:07:25.311424 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:25.311356    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t" podUID="2b453729-cac0-4839-bdfa-0da04e59c1bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Nov 03 18:07:25.312181 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:25.312120    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 03 18:07:25.312290 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:25.312189    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82" podUID="04fd7ef5-cf3c-4ed7-963d-e03294eaf327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 03 18:07:25.321452 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:25.321417    5315 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q" podUID="c974b0f1-e00b-4e4a-b193-df769068983d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:25.413230 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:25.413181    5315 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 03 18:07:25.413703 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:25.413671    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn" podUID="dafbc7c7-538e-4bf6-ab30-970870f21854" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 03 18:07:25.544635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:25.542890    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7" podUID="69760dba-faf9-4c08-895c-ad70fe95fd49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:25.750693 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:25.727790    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr"]
Nov 03 18:07:25.782337 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:25.782309    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r2jgr"]
Nov 03 18:07:25.782537 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:25.757581    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 18:07:25.782696 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:25.782670    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n" podUID="1e4ca152-3632-485f-94b2-4f430cbdfe4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 18:07:25.794495 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:25.794463    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr"
Nov 03 18:07:25.863368 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:25.821356    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-248f8ee3-574c4c5b5-rqjv5" podStartSLOduration=10.821332067 podStartE2EDuration="10.821332067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:07:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:07:24.985197344 +0000 UTC m=+1284.001395438" watchObservedRunningTime="2026-11-03 18:07:25.821332067 +0000 UTC m=+1284.837530136"
Nov 03 18:07:25.864662 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:25.841038    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr" podUID="08159a02-fc2e-4138-824d-be5ba3e141d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:25.904638 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:25.882568    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7"
Nov 03 18:07:25.904638 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:25.884285    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7" podUID="09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:26.024731 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:26.023983    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s" podUID="155e4b39-1877-4bd9-8541-5d6bd3aa0a8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:26.024731 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:26.024377    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt" podUID="7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:26.042160 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:26.042112    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q" podUID="c974b0f1-e00b-4e4a-b193-df769068983d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:26.042795 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:26.042747    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv" podUID="f6d706cb-3aa5-419c-8f6c-29347bb788b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:26.134111 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:26.134065    5315 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7" podUID="69760dba-faf9-4c08-895c-ad70fe95fd49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/health\": context deadline exceeded"
Nov 03 18:07:26.231721 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:26.224681    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c" podUID="8ac59539-8ac1-466e-8d1b-b2be4c42c708" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Nov 03 18:07:26.482634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:26.231325    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 03 18:07:26.482634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:26.468525    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82" podUID="04fd7ef5-cf3c-4ed7-963d-e03294eaf327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 03 18:07:26.482634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:26.255100    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Nov 03 18:07:26.482634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:26.468589    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t" podUID="2b453729-cac0-4839-bdfa-0da04e59c1bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Nov 03 18:07:26.482634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:26.468921    5315 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l" podUID="114dcf54-e4eb-4294-a77b-604fe6d57d48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": context deadline exceeded"
Nov 03 18:07:26.482634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:26.231373    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l" podUID="114dcf54-e4eb-4294-a77b-604fe6d57d48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Nov 03 18:07:26.570631 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:26.548465    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7" podUID="69760dba-faf9-4c08-895c-ad70fe95fd49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:26.602634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:26.571366    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7"
Nov 03 18:07:26.744141 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:26.744032    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 18:07:26.746731 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:26.744380    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n" podUID="1e4ca152-3632-485f-94b2-4f430cbdfe4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 18:07:26.856182 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:26.835886    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-248f8ee3-576cfc4795-5q9nf" event={"ID":"05885de3-0afb-4ae7-a7d2-7c3b64e0bba1","Type":"ContainerStarted","Data":"7862d2ded6fe9289c25dd8dfc0976c69c0083a717ce64057757161dfa64489a1"}
Nov 03 18:07:26.856560 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:26.856507    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr" podUID="df8e0b71-e876-4cea-a7ae-3532c887c660" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:26.907441 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:26.858364    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-248f8ee3-576cfc4795-5q9nf"
Nov 03 18:07:26.948640 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:26.922928    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr" podUID="08159a02-fc2e-4138-824d-be5ba3e141d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:26.948640 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:26.939266    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7" podUID="33725a51-6459-4f15-a866-e9dee41a9732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 03 18:07:27.054630 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:27.046822    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q" podUID="c974b0f1-e00b-4e4a-b193-df769068983d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 18:07:27.262633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:27.048950    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s" podUID="155e4b39-1877-4bd9-8541-5d6bd3aa0a8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:27.263939 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:27.048984    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt" podUID="7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:27.277420 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:27.049340    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv" podUID="f6d706cb-3aa5-419c-8f6c-29347bb788b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:27.277973 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:27.057378    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-248f8ee3-576cfc4795-5q9nf" podStartSLOduration=12.057354132 podStartE2EDuration="12.057354132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:07:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:07:27.055183582 +0000 UTC m=+1286.071381660" watchObservedRunningTime="2026-11-03 18:07:27.057354132 +0000 UTC m=+1286.073552203"
Nov 03 18:07:27.320590 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:27.219156    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Nov 03 18:07:27.320861 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:27.320815    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t" podUID="2b453729-cac0-4839-bdfa-0da04e59c1bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Nov 03 18:07:27.321031 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:27.219204    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 03 18:07:27.321190 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:27.321168    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82" podUID="04fd7ef5-cf3c-4ed7-963d-e03294eaf327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 03 18:07:27.333102 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:27.333069    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f5bff5e-4b15-4cde-a3b0-71c9ae666122" path="/var/lib/kubelet/pods/8f5bff5e-4b15-4cde-a3b0-71c9ae666122/volumes"
Nov 03 18:07:27.348044 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:27.348010    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn"
Nov 03 18:07:27.349377 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:27.349350    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn"
Nov 03 18:07:27.366544 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:27.366516    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn"
Nov 03 18:07:27.479780 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:27.479733    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l" podUID="114dcf54-e4eb-4294-a77b-604fe6d57d48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:27.510545 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:27.510500    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c" podUID="8ac59539-8ac1-466e-8d1b-b2be4c42c708" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:27.549890 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:27.549842    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7" podUID="69760dba-faf9-4c08-895c-ad70fe95fd49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:27.593831 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:27.593728    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7" podUID="09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Nov 03 18:07:27.673634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:27.666790    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l"
Nov 03 18:07:27.673634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:27.668453    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q"
Nov 03 18:07:27.918494 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:27.918355    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr"
Nov 03 18:07:27.921651 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:27.921601    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c"
Nov 03 18:07:27.999634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:27.990935    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv"
Nov 03 18:07:27.999634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:27.945728    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7" podUID="33725a51-6459-4f15-a866-e9dee41a9732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:28.008868 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:28.000756    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7"
Nov 03 18:07:28.134183 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:28.134149    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt"
Nov 03 18:07:28.188410 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:28.188324    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 03 18:07:28.213885 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:28.213833    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82" podUID="04fd7ef5-cf3c-4ed7-963d-e03294eaf327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 03 18:07:28.214154 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:28.194941    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7"
Nov 03 18:07:28.215324 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:28.195209    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Nov 03 18:07:28.215519 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:28.215495    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t" podUID="2b453729-cac0-4839-bdfa-0da04e59c1bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Nov 03 18:07:28.284635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:28.283565    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s" podUID="155e4b39-1877-4bd9-8541-5d6bd3aa0a8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:28.397662 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:28.391291    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s"
Nov 03 18:07:28.399642 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:28.399468    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7"
Nov 03 18:07:28.506694 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:28.504751    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89"]
Nov 03 18:07:28.506694 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:28.505847    5315 cpu_manager.go:414] "containerMap: removing container" podUID="8f5bff5e-4b15-4cde-a3b0-71c9ae666122" containerName="istio-proxy"
Nov 03 18:07:28.506694 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:28.505870    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f5bff5e-4b15-4cde-a3b0-71c9ae666122" containerName="istio-proxy"
Nov 03 18:07:28.506694 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:28.506077    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f5bff5e-4b15-4cde-a3b0-71c9ae666122" containerName="istio-proxy"
Nov 03 18:07:28.604244 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:28.604213    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n"
Nov 03 18:07:28.610649 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:28.606915    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89"
Nov 03 18:07:28.616200 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:28.616169    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89"]
Nov 03 18:07:28.635637 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:28.626037    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n"
Nov 03 18:07:28.652639 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:28.636405    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n"
Nov 03 18:07:28.745643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:28.738374    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-istio-envoy\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89\" (UID: \"57d6511c-f9f2-4ca7-9bfb-6475dd17f94b\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89"
Nov 03 18:07:28.745643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:28.738438    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-istio-token\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89\" (UID: \"57d6511c-f9f2-4ca7-9bfb-6475dd17f94b\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89"
Nov 03 18:07:28.745643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:28.738550    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vsft\" (UniqueName: \"kubernetes.io/projected/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-kube-api-access-2vsft\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89\" (UID: \"57d6511c-f9f2-4ca7-9bfb-6475dd17f94b\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89"
Nov 03 18:07:28.745643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:28.738591    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-istio-data\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89\" (UID: \"57d6511c-f9f2-4ca7-9bfb-6475dd17f94b\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89"
Nov 03 18:07:28.745643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:28.738688    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-istiod-ca-cert\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89\" (UID: \"57d6511c-f9f2-4ca7-9bfb-6475dd17f94b\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89"
Nov 03 18:07:28.745643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:28.738871    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-workload-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89\" (UID: \"57d6511c-f9f2-4ca7-9bfb-6475dd17f94b\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89"
Nov 03 18:07:28.745643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:28.738915    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-istio-podinfo\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89\" (UID: \"57d6511c-f9f2-4ca7-9bfb-6475dd17f94b\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89"
Nov 03 18:07:28.745643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:28.739009    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-workload-certs\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89\" (UID: \"57d6511c-f9f2-4ca7-9bfb-6475dd17f94b\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89"
Nov 03 18:07:28.745643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:28.739057    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-credential-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89\" (UID: \"57d6511c-f9f2-4ca7-9bfb-6475dd17f94b\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89"
Nov 03 18:07:28.842147 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:28.842101    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-istiod-ca-cert\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89\" (UID: \"57d6511c-f9f2-4ca7-9bfb-6475dd17f94b\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89"
Nov 03 18:07:28.849633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:28.842532    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-workload-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89\" (UID: \"57d6511c-f9f2-4ca7-9bfb-6475dd17f94b\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89"
Nov 03 18:07:28.850096 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:28.850064    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-istio-podinfo\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89\" (UID: \"57d6511c-f9f2-4ca7-9bfb-6475dd17f94b\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89"
Nov 03 18:07:28.871672 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:28.853117    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-istiod-ca-cert\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89\" (UID: \"57d6511c-f9f2-4ca7-9bfb-6475dd17f94b\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89"
Nov 03 18:07:28.932654 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:28.869669    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-workload-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89\" (UID: \"57d6511c-f9f2-4ca7-9bfb-6475dd17f94b\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89"
Nov 03 18:07:28.932654 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:28.892859    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-workload-certs\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89\" (UID: \"57d6511c-f9f2-4ca7-9bfb-6475dd17f94b\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89"
Nov 03 18:07:28.932654 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:28.892922    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-credential-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89\" (UID: \"57d6511c-f9f2-4ca7-9bfb-6475dd17f94b\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89"
Nov 03 18:07:28.932654 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:28.893072    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-istio-envoy\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89\" (UID: \"57d6511c-f9f2-4ca7-9bfb-6475dd17f94b\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89"
Nov 03 18:07:28.932654 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:28.893113    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-istio-token\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89\" (UID: \"57d6511c-f9f2-4ca7-9bfb-6475dd17f94b\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89"
Nov 03 18:07:28.932654 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:28.893238    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vsft\" (UniqueName: \"kubernetes.io/projected/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-kube-api-access-2vsft\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89\" (UID: \"57d6511c-f9f2-4ca7-9bfb-6475dd17f94b\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89"
Nov 03 18:07:28.932654 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:28.893280    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-istio-data\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89\" (UID: \"57d6511c-f9f2-4ca7-9bfb-6475dd17f94b\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89"
Nov 03 18:07:28.932654 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:28.893286    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-workload-certs\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89\" (UID: \"57d6511c-f9f2-4ca7-9bfb-6475dd17f94b\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89"
Nov 03 18:07:28.932654 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:28.893553    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-istio-data\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89\" (UID: \"57d6511c-f9f2-4ca7-9bfb-6475dd17f94b\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89"
Nov 03 18:07:28.932654 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:28.894427    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-credential-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89\" (UID: \"57d6511c-f9f2-4ca7-9bfb-6475dd17f94b\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89"
Nov 03 18:07:29.191630 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:29.191530    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 03 18:07:29.191903 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:29.191866    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82" podUID="04fd7ef5-cf3c-4ed7-963d-e03294eaf327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 03 18:07:29.194736 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:29.194709    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Nov 03 18:07:29.194934 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:29.194909    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t" podUID="2b453729-cac0-4839-bdfa-0da04e59c1bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Nov 03 18:07:29.305683 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:29.301242    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82"
Nov 03 18:07:29.305683 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:29.301328    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t"
Nov 03 18:07:29.379897 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:29.379357    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vsft\" (UniqueName: \"kubernetes.io/projected/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-kube-api-access-2vsft\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89\" (UID: \"57d6511c-f9f2-4ca7-9bfb-6475dd17f94b\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89"
Nov 03 18:07:29.562419 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:29.515845    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-istio-token\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89\" (UID: \"57d6511c-f9f2-4ca7-9bfb-6475dd17f94b\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89"
Nov 03 18:07:29.686645 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:29.563658    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv" podUID="f6d706cb-3aa5-419c-8f6c-29347bb788b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:29.686645 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:29.569075    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-istio-envoy\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89\" (UID: \"57d6511c-f9f2-4ca7-9bfb-6475dd17f94b\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89"
Nov 03 18:07:29.703631 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:29.574286    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e49abb84-phtcf"]
Nov 03 18:07:29.703631 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:29.643517    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s" podUID="155e4b39-1877-4bd9-8541-5d6bd3aa0a8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:29.759221 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:29.759175    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-istio-podinfo\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89\" (UID: \"57d6511c-f9f2-4ca7-9bfb-6475dd17f94b\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89"
Nov 03 18:07:29.794981 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:29.794698    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e49abb84-phtcf"]
Nov 03 18:07:29.794981 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:29.794748    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4b6e7259-54b9769cbc-k5qbr"]
Nov 03 18:07:29.796203 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:29.796166    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89"
Nov 03 18:07:29.798962 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:29.798938    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e49abb84-phtcf"
Nov 03 18:07:29.802147 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:29.801584    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4b6e7259-54b9769cbc-k5qbr"
Nov 03 18:07:29.804661 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:29.804639    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4b6e7259-54b9769cbc-k5qbr"]
Nov 03 18:07:29.818605 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:29.818584    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4b6e7259-854d546fff-j5qbr"]
Nov 03 18:07:29.826164 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:29.826139    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b6e7259-854d546fff-j5qbr"
Nov 03 18:07:29.832896 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:29.832408    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4b6e7259-854d546fff-j5qbr"]
Nov 03 18:07:29.848544 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:29.848495    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:07:29.848544 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:29.848496    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:07:29.848544 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:29.848529    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:07:29.848818 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:29.848559    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:07:29.849594 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:29.849569    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4b6e7259\""
Nov 03 18:07:29.948992 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:29.948960    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4b6e7259-dockercfg-kdnnb\""
Nov 03 18:07:29.984715 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:29.984656    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dd7528b-e815-44b0-aa17-72f16301468e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4b6e7259-854d546fff-j5qbr\" (UID: \"1dd7528b-e815-44b0-aa17-72f16301468e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4b6e7259-854d546fff-j5qbr"
Nov 03 18:07:29.985872 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:29.984782    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrp28\" (UniqueName: \"kubernetes.io/projected/1dd7528b-e815-44b0-aa17-72f16301468e-kube-api-access-mrp28\") pod \"payload-pre-processing-e2e-ait-4b6e7259-854d546fff-j5qbr\" (UID: \"1dd7528b-e815-44b0-aa17-72f16301468e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4b6e7259-854d546fff-j5qbr"
Nov 03 18:07:29.985872 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:29.984816    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zmcq\" (UniqueName: \"kubernetes.io/projected/ccbbd645-e40b-4d27-8b93-e6d91a2d0184-kube-api-access-8zmcq\") pod \"maas-api-revoke-sub-e49abb84-phtcf\" (UID: \"ccbbd645-e40b-4d27-8b93-e6d91a2d0184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e49abb84-phtcf"
Nov 03 18:07:29.985872 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:29.984862    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwhp2\" (UniqueName: \"kubernetes.io/projected/fdafb581-457b-4a16-a9b7-9c19f90f30cd-kube-api-access-rwhp2\") pod \"payload-processing-e2e-ait-4b6e7259-54b9769cbc-k5qbr\" (UID: \"fdafb581-457b-4a16-a9b7-9c19f90f30cd\") " pod="openshift-ingress/payload-processing-e2e-ait-4b6e7259-54b9769cbc-k5qbr"
Nov 03 18:07:29.985872 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:29.985024    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccbbd645-e40b-4d27-8b93-e6d91a2d0184-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e49abb84-phtcf\" (UID: \"ccbbd645-e40b-4d27-8b93-e6d91a2d0184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e49abb84-phtcf"
Nov 03 18:07:29.985872 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:29.985114    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdafb581-457b-4a16-a9b7-9c19f90f30cd-plugins-config-volume\") pod \"payload-processing-e2e-ait-4b6e7259-54b9769cbc-k5qbr\" (UID: \"fdafb581-457b-4a16-a9b7-9c19f90f30cd\") " pod="openshift-ingress/payload-processing-e2e-ait-4b6e7259-54b9769cbc-k5qbr"
Nov 03 18:07:29.985872 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:29.985162    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccbbd645-e40b-4d27-8b93-e6d91a2d0184-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e49abb84-phtcf\" (UID: \"ccbbd645-e40b-4d27-8b93-e6d91a2d0184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e49abb84-phtcf"
Nov 03 18:07:30.086909 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:30.086798    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrp28\" (UniqueName: \"kubernetes.io/projected/1dd7528b-e815-44b0-aa17-72f16301468e-kube-api-access-mrp28\") pod \"payload-pre-processing-e2e-ait-4b6e7259-854d546fff-j5qbr\" (UID: \"1dd7528b-e815-44b0-aa17-72f16301468e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4b6e7259-854d546fff-j5qbr"
Nov 03 18:07:30.086909 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:30.086856    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zmcq\" (UniqueName: \"kubernetes.io/projected/ccbbd645-e40b-4d27-8b93-e6d91a2d0184-kube-api-access-8zmcq\") pod \"maas-api-revoke-sub-e49abb84-phtcf\" (UID: \"ccbbd645-e40b-4d27-8b93-e6d91a2d0184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e49abb84-phtcf"
Nov 03 18:07:30.086909 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:30.086896    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwhp2\" (UniqueName: \"kubernetes.io/projected/fdafb581-457b-4a16-a9b7-9c19f90f30cd-kube-api-access-rwhp2\") pod \"payload-processing-e2e-ait-4b6e7259-54b9769cbc-k5qbr\" (UID: \"fdafb581-457b-4a16-a9b7-9c19f90f30cd\") " pod="openshift-ingress/payload-processing-e2e-ait-4b6e7259-54b9769cbc-k5qbr"
Nov 03 18:07:30.087203 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:30.087016    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccbbd645-e40b-4d27-8b93-e6d91a2d0184-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e49abb84-phtcf\" (UID: \"ccbbd645-e40b-4d27-8b93-e6d91a2d0184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e49abb84-phtcf"
Nov 03 18:07:30.087203 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:30.087056    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdafb581-457b-4a16-a9b7-9c19f90f30cd-plugins-config-volume\") pod \"payload-processing-e2e-ait-4b6e7259-54b9769cbc-k5qbr\" (UID: \"fdafb581-457b-4a16-a9b7-9c19f90f30cd\") " pod="openshift-ingress/payload-processing-e2e-ait-4b6e7259-54b9769cbc-k5qbr"
Nov 03 18:07:30.087203 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:30.087120    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccbbd645-e40b-4d27-8b93-e6d91a2d0184-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e49abb84-phtcf\" (UID: \"ccbbd645-e40b-4d27-8b93-e6d91a2d0184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e49abb84-phtcf"
Nov 03 18:07:30.087369 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:30.087265    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dd7528b-e815-44b0-aa17-72f16301468e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4b6e7259-854d546fff-j5qbr\" (UID: \"1dd7528b-e815-44b0-aa17-72f16301468e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4b6e7259-854d546fff-j5qbr"
Nov 03 18:07:30.088314 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:30.088287    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdafb581-457b-4a16-a9b7-9c19f90f30cd-plugins-config-volume\") pod \"payload-processing-e2e-ait-4b6e7259-54b9769cbc-k5qbr\" (UID: \"fdafb581-457b-4a16-a9b7-9c19f90f30cd\") " pod="openshift-ingress/payload-processing-e2e-ait-4b6e7259-54b9769cbc-k5qbr"
Nov 03 18:07:30.088314 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:30.088336    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dd7528b-e815-44b0-aa17-72f16301468e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4b6e7259-854d546fff-j5qbr\" (UID: \"1dd7528b-e815-44b0-aa17-72f16301468e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4b6e7259-854d546fff-j5qbr"
Nov 03 18:07:30.088554 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:30.088522    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccbbd645-e40b-4d27-8b93-e6d91a2d0184-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e49abb84-phtcf\" (UID: \"ccbbd645-e40b-4d27-8b93-e6d91a2d0184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e49abb84-phtcf"
Nov 03 18:07:30.090471 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:30.090444    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccbbd645-e40b-4d27-8b93-e6d91a2d0184-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e49abb84-phtcf\" (UID: \"ccbbd645-e40b-4d27-8b93-e6d91a2d0184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e49abb84-phtcf"
Nov 03 18:07:30.146425 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:30.146390    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zmcq\" (UniqueName: \"kubernetes.io/projected/ccbbd645-e40b-4d27-8b93-e6d91a2d0184-kube-api-access-8zmcq\") pod \"maas-api-revoke-sub-e49abb84-phtcf\" (UID: \"ccbbd645-e40b-4d27-8b93-e6d91a2d0184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e49abb84-phtcf"
Nov 03 18:07:30.147410 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:30.147383    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrp28\" (UniqueName: \"kubernetes.io/projected/1dd7528b-e815-44b0-aa17-72f16301468e-kube-api-access-mrp28\") pod \"payload-pre-processing-e2e-ait-4b6e7259-854d546fff-j5qbr\" (UID: \"1dd7528b-e815-44b0-aa17-72f16301468e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4b6e7259-854d546fff-j5qbr"
Nov 03 18:07:30.148773 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:30.148746    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwhp2\" (UniqueName: \"kubernetes.io/projected/fdafb581-457b-4a16-a9b7-9c19f90f30cd-kube-api-access-rwhp2\") pod \"payload-processing-e2e-ait-4b6e7259-54b9769cbc-k5qbr\" (UID: \"fdafb581-457b-4a16-a9b7-9c19f90f30cd\") " pod="openshift-ingress/payload-processing-e2e-ait-4b6e7259-54b9769cbc-k5qbr"
Nov 03 18:07:30.151051 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:30.150845    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e49abb84-phtcf"
Nov 03 18:07:30.173935 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:30.173899    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 03 18:07:30.174083 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:30.173967    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82" podUID="04fd7ef5-cf3c-4ed7-963d-e03294eaf327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 03 18:07:30.186433 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:30.186399    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4b6e7259-54b9769cbc-k5qbr"
Nov 03 18:07:30.189806 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:30.189694    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Nov 03 18:07:30.189806 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:30.189746    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t" podUID="2b453729-cac0-4839-bdfa-0da04e59c1bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Nov 03 18:07:30.204820 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:30.204787    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b6e7259-854d546fff-j5qbr"
Nov 03 18:07:30.287545 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:30.287431    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89"]
Nov 03 18:07:30.314872 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:30.314501    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:07:30.314872 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:30.314572    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:07:30.314872 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:30.314599    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:07:30.622824 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:30.622353    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e49abb84-phtcf"]
Nov 03 18:07:30.939656 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:30.939582    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4b6e7259-54b9769cbc-k5qbr"]
Nov 03 18:07:30.949004 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:30.948953    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4b6e7259-854d546fff-j5qbr"]
Nov 03 18:07:30.952819 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:07:30.952685    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dd7528b_e815_44b0_aa17_72f16301468e.slice/crio-ecdcbcf6477d4107d221c10daa5b3a352cbcb4455437c8f00f8dc656bf6fe9a7 WatchSource:0}: Error finding container ecdcbcf6477d4107d221c10daa5b3a352cbcb4455437c8f00f8dc656bf6fe9a7: Status 404 returned error can't find the container with id ecdcbcf6477d4107d221c10daa5b3a352cbcb4455437c8f00f8dc656bf6fe9a7
Nov 03 18:07:31.286053 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:31.286008    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b6e7259-854d546fff-j5qbr" event={"ID":"1dd7528b-e815-44b0-aa17-72f16301468e","Type":"ContainerStarted","Data":"ecdcbcf6477d4107d221c10daa5b3a352cbcb4455437c8f00f8dc656bf6fe9a7"}
Nov 03 18:07:31.288092 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:31.288059    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4b6e7259-54b9769cbc-k5qbr" event={"ID":"fdafb581-457b-4a16-a9b7-9c19f90f30cd","Type":"ContainerStarted","Data":"85c69d2a5c307c07c63b3712a60118844b6cd3c130aa6d0f824bb4fc0001d27d"}
Nov 03 18:07:31.292268 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:31.292240    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89" event={"ID":"57d6511c-f9f2-4ca7-9bfb-6475dd17f94b","Type":"ContainerStarted","Data":"dbe840016f3b77963094a5c29481f8902ccdbaba738663cc148f0f91459ca3f7"}
Nov 03 18:07:31.292447 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:31.292430    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89" event={"ID":"57d6511c-f9f2-4ca7-9bfb-6475dd17f94b","Type":"ContainerStarted","Data":"c4afd48ae10bf1cc64a9ea1744c1308935e30b2df494ed6d17526328607d81e2"}
Nov 03 18:07:31.293767 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:31.293743    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e49abb84-phtcf" event={"ID":"ccbbd645-e40b-4d27-8b93-e6d91a2d0184","Type":"ContainerStarted","Data":"e080875ab71b031d3f8acb7fdbf25a353cc6d9067839f3b469c60548edefd464"}
Nov 03 18:07:31.411777 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:31.409288    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89" podStartSLOduration=3.409268886 podStartE2EDuration="3.409268886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:07:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:07:31.406832021 +0000 UTC m=+1290.423030093" watchObservedRunningTime="2026-11-03 18:07:31.409268886 +0000 UTC m=+1290.425466956"
Nov 03 18:07:31.622260 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:31.622226    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t"
Nov 03 18:07:31.624304 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:31.624277    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t"
Nov 03 18:07:31.798110 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:31.798074    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89"
Nov 03 18:07:31.799311 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:31.799281    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Nov 03 18:07:31.799439 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:31.799371    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89" podUID="57d6511c-f9f2-4ca7-9bfb-6475dd17f94b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Nov 03 18:07:31.801885 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:31.801845    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82"
Nov 03 18:07:31.803296 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:31.803274    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82"
Nov 03 18:07:32.304208 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:32.304175    5315 generic.go:356] "Generic (PLEG): container finished" podID="ccbbd645-e40b-4d27-8b93-e6d91a2d0184" containerID="435362a644b1ee21b1ed74f3effb3cec52f9c2a1effc551b50d7c54542719e05" exitCode=0
Nov 03 18:07:32.304600 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:32.304260    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e49abb84-phtcf" event={"ID":"ccbbd645-e40b-4d27-8b93-e6d91a2d0184","Type":"ContainerDied","Data":"435362a644b1ee21b1ed74f3effb3cec52f9c2a1effc551b50d7c54542719e05"}
Nov 03 18:07:32.306004 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:32.305975    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b6e7259-854d546fff-j5qbr" event={"ID":"1dd7528b-e815-44b0-aa17-72f16301468e","Type":"ContainerStarted","Data":"270400b33a1623b292a8a153d05f6c79d21632ded1f82c8fba0f00ba3e356520"}
Nov 03 18:07:32.306096 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:32.306032    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b6e7259-854d546fff-j5qbr"
Nov 03 18:07:32.308177 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:32.308145    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4b6e7259-54b9769cbc-k5qbr" event={"ID":"fdafb581-457b-4a16-a9b7-9c19f90f30cd","Type":"ContainerStarted","Data":"543ad04eb72d611013219d1ef3907b5a3998c949737db9f8458c8243d92ed4fc"}
Nov 03 18:07:32.308640 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:32.308603    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-4b6e7259-54b9769cbc-k5qbr"
Nov 03 18:07:32.561808 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:32.561750    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b6e7259-854d546fff-j5qbr" podStartSLOduration=3.561734602 podStartE2EDuration="3.561734602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:07:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:07:32.525465796 +0000 UTC m=+1291.541663866" watchObservedRunningTime="2026-11-03 18:07:32.561734602 +0000 UTC m=+1291.577932670"
Nov 03 18:07:32.805450 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:32.800245    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Nov 03 18:07:32.805450 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:32.800338    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89" podUID="57d6511c-f9f2-4ca7-9bfb-6475dd17f94b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Nov 03 18:07:33.776123 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:33.776094    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e49abb84-phtcf"
Nov 03 18:07:33.798684 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:33.798652    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Nov 03 18:07:33.798817 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:33.798700    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89" podUID="57d6511c-f9f2-4ca7-9bfb-6475dd17f94b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Nov 03 18:07:33.911130 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:33.911028    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-4b6e7259-54b9769cbc-k5qbr" podStartSLOduration=4.911014756 podStartE2EDuration="4.911014756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:07:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:07:32.661921355 +0000 UTC m=+1291.678119426" watchObservedRunningTime="2026-11-03 18:07:33.911014756 +0000 UTC m=+1292.927212825"
Nov 03 18:07:33.972699 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:33.972642    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccbbd645-e40b-4d27-8b93-e6d91a2d0184-kube-api-access-8zmcq\" (UniqueName: \"kubernetes.io/projected/ccbbd645-e40b-4d27-8b93-e6d91a2d0184-kube-api-access-8zmcq\") pod \"ccbbd645-e40b-4d27-8b93-e6d91a2d0184\" (UID: \"ccbbd645-e40b-4d27-8b93-e6d91a2d0184\") "
Nov 03 18:07:33.972903 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:33.972809    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccbbd645-e40b-4d27-8b93-e6d91a2d0184-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccbbd645-e40b-4d27-8b93-e6d91a2d0184-maas-api-openshift-ca\") pod \"ccbbd645-e40b-4d27-8b93-e6d91a2d0184\" (UID: \"ccbbd645-e40b-4d27-8b93-e6d91a2d0184\") "
Nov 03 18:07:33.972903 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:33.972870    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccbbd645-e40b-4d27-8b93-e6d91a2d0184-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccbbd645-e40b-4d27-8b93-e6d91a2d0184-maas-api-xks-ca\") pod \"ccbbd645-e40b-4d27-8b93-e6d91a2d0184\" (UID: \"ccbbd645-e40b-4d27-8b93-e6d91a2d0184\") "
Nov 03 18:07:33.973523 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:33.973491    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccbbd645-e40b-4d27-8b93-e6d91a2d0184-maas-api-openshift-ca" pod "ccbbd645-e40b-4d27-8b93-e6d91a2d0184" (UID: "ccbbd645-e40b-4d27-8b93-e6d91a2d0184"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:07:33.975312 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:33.975285    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccbbd645-e40b-4d27-8b93-e6d91a2d0184-maas-api-xks-ca" pod "ccbbd645-e40b-4d27-8b93-e6d91a2d0184" (UID: "ccbbd645-e40b-4d27-8b93-e6d91a2d0184"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:07:33.975387 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:33.975357    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccbbd645-e40b-4d27-8b93-e6d91a2d0184-kube-api-access-8zmcq" pod "ccbbd645-e40b-4d27-8b93-e6d91a2d0184" (UID: "ccbbd645-e40b-4d27-8b93-e6d91a2d0184"). InnerVolumeSpecName "kube-api-access-8zmcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:07:34.074777 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:34.074734    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccbbd645-e40b-4d27-8b93-e6d91a2d0184-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:07:34.074777 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:34.074774    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zmcq\" (UniqueName: \"kubernetes.io/projected/ccbbd645-e40b-4d27-8b93-e6d91a2d0184-kube-api-access-8zmcq\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:07:34.075030 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:34.074789    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccbbd645-e40b-4d27-8b93-e6d91a2d0184-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:07:34.351314 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:34.351278    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e49abb84-phtcf"
Nov 03 18:07:34.351314 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:34.351299    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e49abb84-phtcf" event={"ID":"ccbbd645-e40b-4d27-8b93-e6d91a2d0184","Type":"ContainerDied","Data":"e080875ab71b031d3f8acb7fdbf25a353cc6d9067839f3b469c60548edefd464"}
Nov 03 18:07:34.351573 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:34.351361    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e080875ab71b031d3f8acb7fdbf25a353cc6d9067839f3b469c60548edefd464"
Nov 03 18:07:34.797823 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:34.797778    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Nov 03 18:07:34.798343 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:34.797866    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89" podUID="57d6511c-f9f2-4ca7-9bfb-6475dd17f94b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Nov 03 18:07:35.798099 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:35.798055    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Nov 03 18:07:35.799329 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:35.798133    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89" podUID="57d6511c-f9f2-4ca7-9bfb-6475dd17f94b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Nov 03 18:07:36.808635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:36.804521    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Nov 03 18:07:36.808635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:36.804628    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89" podUID="57d6511c-f9f2-4ca7-9bfb-6475dd17f94b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Nov 03 18:07:36.875634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:36.874071    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-248f8ee3-574c4c5b5-rqjv5"
Nov 03 18:07:37.798651 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:37.798589    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Nov 03 18:07:37.798833 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:37.798693    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89" podUID="57d6511c-f9f2-4ca7-9bfb-6475dd17f94b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Nov 03 18:07:39.125641 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:39.125572    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89"
Nov 03 18:07:39.126357 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:39.126329    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89"
Nov 03 18:07:39.127866 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:39.127838    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89"
Nov 03 18:07:39.193869 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:39.193839    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-248f8ee3-576cfc4795-5q9nf"
Nov 03 18:07:39.644544 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:39.640150    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-727fe8c5-gtqh9"]
Nov 03 18:07:39.647403 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:39.647366    5315 cpu_manager.go:414] "containerMap: removing container" podUID="ccbbd645-e40b-4d27-8b93-e6d91a2d0184" containerName="revoke-keys"
Nov 03 18:07:39.647403 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:39.647403    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccbbd645-e40b-4d27-8b93-e6d91a2d0184" containerName="revoke-keys"
Nov 03 18:07:39.647683 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:39.647641    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccbbd645-e40b-4d27-8b93-e6d91a2d0184" containerName="revoke-keys"
Nov 03 18:07:39.651699 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:39.651677    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-727fe8c5-gtqh9"
Nov 03 18:07:39.701903 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:39.701870    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:07:39.701903 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:39.701892    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:07:39.702329 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:39.702295    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:07:39.702463 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:39.702392    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:07:39.710090 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:39.710059    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-727fe8c5-gtqh9"]
Nov 03 18:07:39.728764 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:39.728720    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b48dcdb-9a6d-4e03-854f-ece61e03990e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-727fe8c5-gtqh9\" (UID: \"6b48dcdb-9a6d-4e03-854f-ece61e03990e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-727fe8c5-gtqh9"
Nov 03 18:07:39.728945 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:39.728833    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrwr2\" (UniqueName: \"kubernetes.io/projected/6b48dcdb-9a6d-4e03-854f-ece61e03990e-kube-api-access-vrwr2\") pod \"maas-api-revoke-sub-727fe8c5-gtqh9\" (UID: \"6b48dcdb-9a6d-4e03-854f-ece61e03990e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-727fe8c5-gtqh9"
Nov 03 18:07:39.729090 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:39.729067    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b48dcdb-9a6d-4e03-854f-ece61e03990e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-727fe8c5-gtqh9\" (UID: \"6b48dcdb-9a6d-4e03-854f-ece61e03990e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-727fe8c5-gtqh9"
Nov 03 18:07:39.830355 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:39.830312    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b48dcdb-9a6d-4e03-854f-ece61e03990e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-727fe8c5-gtqh9\" (UID: \"6b48dcdb-9a6d-4e03-854f-ece61e03990e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-727fe8c5-gtqh9"
Nov 03 18:07:39.830565 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:39.830470    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b48dcdb-9a6d-4e03-854f-ece61e03990e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-727fe8c5-gtqh9\" (UID: \"6b48dcdb-9a6d-4e03-854f-ece61e03990e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-727fe8c5-gtqh9"
Nov 03 18:07:39.830666 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:39.830563    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrwr2\" (UniqueName: \"kubernetes.io/projected/6b48dcdb-9a6d-4e03-854f-ece61e03990e-kube-api-access-vrwr2\") pod \"maas-api-revoke-sub-727fe8c5-gtqh9\" (UID: \"6b48dcdb-9a6d-4e03-854f-ece61e03990e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-727fe8c5-gtqh9"
Nov 03 18:07:39.831235 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:39.831213    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b48dcdb-9a6d-4e03-854f-ece61e03990e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-727fe8c5-gtqh9\" (UID: \"6b48dcdb-9a6d-4e03-854f-ece61e03990e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-727fe8c5-gtqh9"
Nov 03 18:07:39.832831 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:39.832810    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b48dcdb-9a6d-4e03-854f-ece61e03990e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-727fe8c5-gtqh9\" (UID: \"6b48dcdb-9a6d-4e03-854f-ece61e03990e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-727fe8c5-gtqh9"
Nov 03 18:07:39.888358 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:39.888290    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrwr2\" (UniqueName: \"kubernetes.io/projected/6b48dcdb-9a6d-4e03-854f-ece61e03990e-kube-api-access-vrwr2\") pod \"maas-api-revoke-sub-727fe8c5-gtqh9\" (UID: \"6b48dcdb-9a6d-4e03-854f-ece61e03990e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-727fe8c5-gtqh9"
Nov 03 18:07:39.966927 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:39.966836    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-727fe8c5-gtqh9"
Nov 03 18:07:40.386374 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:40.386343    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-727fe8c5-gtqh9"]
Nov 03 18:07:40.463312 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:40.463273    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-727fe8c5-gtqh9" event={"ID":"6b48dcdb-9a6d-4e03-854f-ece61e03990e","Type":"ContainerStarted","Data":"5b71d5e1ea9ee7615f9823e6be0e7ccd38585e4422601d3213efb28cf1ede1ab"}
Nov 03 18:07:41.473558 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:41.473519    5315 generic.go:356] "Generic (PLEG): container finished" podID="6b48dcdb-9a6d-4e03-854f-ece61e03990e" containerID="1898f53c3072938b45dd415a66459597fa9290d6524987c75ead57c10671034f" exitCode=0
Nov 03 18:07:41.474089 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:41.473625    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-727fe8c5-gtqh9" event={"ID":"6b48dcdb-9a6d-4e03-854f-ece61e03990e","Type":"ContainerDied","Data":"1898f53c3072938b45dd415a66459597fa9290d6524987c75ead57c10671034f"}
Nov 03 18:07:42.564174 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:42.564140    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69"]
Nov 03 18:07:42.572191 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:42.572145    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69"
Nov 03 18:07:42.654351 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:42.654320    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-727fe8c5-gtqh9"
Nov 03 18:07:42.676368 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:42.675192    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69"]
Nov 03 18:07:42.724825 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:42.724781    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1bcb2fe-aef1-446a-9e93-a18dc824d388-workload-certs\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69\" (UID: \"a1bcb2fe-aef1-446a-9e93-a18dc824d388\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69"
Nov 03 18:07:42.725068 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:42.725040    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1bcb2fe-aef1-446a-9e93-a18dc824d388-istio-podinfo\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69\" (UID: \"a1bcb2fe-aef1-446a-9e93-a18dc824d388\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69"
Nov 03 18:07:42.725202 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:42.725187    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bcb2fe-aef1-446a-9e93-a18dc824d388-workload-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69\" (UID: \"a1bcb2fe-aef1-446a-9e93-a18dc824d388\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69"
Nov 03 18:07:42.725270 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:42.725216    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1bcb2fe-aef1-446a-9e93-a18dc824d388-istio-data\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69\" (UID: \"a1bcb2fe-aef1-446a-9e93-a18dc824d388\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69"
Nov 03 18:07:42.725270 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:42.725253    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bcb2fe-aef1-446a-9e93-a18dc824d388-credential-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69\" (UID: \"a1bcb2fe-aef1-446a-9e93-a18dc824d388\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69"
Nov 03 18:07:42.725412 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:42.725391    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c6r4\" (UniqueName: \"kubernetes.io/projected/a1bcb2fe-aef1-446a-9e93-a18dc824d388-kube-api-access-9c6r4\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69\" (UID: \"a1bcb2fe-aef1-446a-9e93-a18dc824d388\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69"
Nov 03 18:07:42.725450 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:42.725431    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1bcb2fe-aef1-446a-9e93-a18dc824d388-istio-token\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69\" (UID: \"a1bcb2fe-aef1-446a-9e93-a18dc824d388\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69"
Nov 03 18:07:42.725554 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:42.725536    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1bcb2fe-aef1-446a-9e93-a18dc824d388-istiod-ca-cert\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69\" (UID: \"a1bcb2fe-aef1-446a-9e93-a18dc824d388\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69"
Nov 03 18:07:42.725645 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:42.725623    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1bcb2fe-aef1-446a-9e93-a18dc824d388-istio-envoy\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69\" (UID: \"a1bcb2fe-aef1-446a-9e93-a18dc824d388\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69"
Nov 03 18:07:42.826780 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:42.826735    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b48dcdb-9a6d-4e03-854f-ece61e03990e-kube-api-access-vrwr2\" (UniqueName: \"kubernetes.io/projected/6b48dcdb-9a6d-4e03-854f-ece61e03990e-kube-api-access-vrwr2\") pod \"6b48dcdb-9a6d-4e03-854f-ece61e03990e\" (UID: \"6b48dcdb-9a6d-4e03-854f-ece61e03990e\") "
Nov 03 18:07:42.827059 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:42.827033    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b48dcdb-9a6d-4e03-854f-ece61e03990e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b48dcdb-9a6d-4e03-854f-ece61e03990e-maas-api-openshift-ca\") pod \"6b48dcdb-9a6d-4e03-854f-ece61e03990e\" (UID: \"6b48dcdb-9a6d-4e03-854f-ece61e03990e\") "
Nov 03 18:07:42.827153 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:42.827118    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b48dcdb-9a6d-4e03-854f-ece61e03990e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b48dcdb-9a6d-4e03-854f-ece61e03990e-maas-api-xks-ca\") pod \"6b48dcdb-9a6d-4e03-854f-ece61e03990e\" (UID: \"6b48dcdb-9a6d-4e03-854f-ece61e03990e\") "
Nov 03 18:07:42.827547 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:42.827520    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1bcb2fe-aef1-446a-9e93-a18dc824d388-istio-podinfo\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69\" (UID: \"a1bcb2fe-aef1-446a-9e93-a18dc824d388\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69"
Nov 03 18:07:42.827692 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:42.827591    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b48dcdb-9a6d-4e03-854f-ece61e03990e-maas-api-openshift-ca" pod "6b48dcdb-9a6d-4e03-854f-ece61e03990e" (UID: "6b48dcdb-9a6d-4e03-854f-ece61e03990e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:07:42.827692 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:42.827672    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bcb2fe-aef1-446a-9e93-a18dc824d388-workload-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69\" (UID: \"a1bcb2fe-aef1-446a-9e93-a18dc824d388\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69"
Nov 03 18:07:42.827825 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:42.827699    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1bcb2fe-aef1-446a-9e93-a18dc824d388-istio-data\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69\" (UID: \"a1bcb2fe-aef1-446a-9e93-a18dc824d388\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69"
Nov 03 18:07:42.827825 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:42.827730    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bcb2fe-aef1-446a-9e93-a18dc824d388-credential-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69\" (UID: \"a1bcb2fe-aef1-446a-9e93-a18dc824d388\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69"
Nov 03 18:07:42.827825 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:42.827762    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c6r4\" (UniqueName: \"kubernetes.io/projected/a1bcb2fe-aef1-446a-9e93-a18dc824d388-kube-api-access-9c6r4\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69\" (UID: \"a1bcb2fe-aef1-446a-9e93-a18dc824d388\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69"
Nov 03 18:07:42.827825 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:42.827791    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1bcb2fe-aef1-446a-9e93-a18dc824d388-istio-token\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69\" (UID: \"a1bcb2fe-aef1-446a-9e93-a18dc824d388\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69"
Nov 03 18:07:42.828036 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:42.827857    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1bcb2fe-aef1-446a-9e93-a18dc824d388-istiod-ca-cert\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69\" (UID: \"a1bcb2fe-aef1-446a-9e93-a18dc824d388\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69"
Nov 03 18:07:42.828036 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:42.827920    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1bcb2fe-aef1-446a-9e93-a18dc824d388-istio-envoy\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69\" (UID: \"a1bcb2fe-aef1-446a-9e93-a18dc824d388\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69"
Nov 03 18:07:42.828036 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:42.827989    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1bcb2fe-aef1-446a-9e93-a18dc824d388-workload-certs\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69\" (UID: \"a1bcb2fe-aef1-446a-9e93-a18dc824d388\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69"
Nov 03 18:07:42.828170 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:42.828085    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b48dcdb-9a6d-4e03-854f-ece61e03990e-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:07:42.828170 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:42.828120    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bcb2fe-aef1-446a-9e93-a18dc824d388-workload-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69\" (UID: \"a1bcb2fe-aef1-446a-9e93-a18dc824d388\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69"
Nov 03 18:07:42.828336 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:42.828218    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1bcb2fe-aef1-446a-9e93-a18dc824d388-istio-data\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69\" (UID: \"a1bcb2fe-aef1-446a-9e93-a18dc824d388\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69"
Nov 03 18:07:42.828407 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:42.828341    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1bcb2fe-aef1-446a-9e93-a18dc824d388-workload-certs\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69\" (UID: \"a1bcb2fe-aef1-446a-9e93-a18dc824d388\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69"
Nov 03 18:07:42.828754 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:42.828722    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1bcb2fe-aef1-446a-9e93-a18dc824d388-istiod-ca-cert\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69\" (UID: \"a1bcb2fe-aef1-446a-9e93-a18dc824d388\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69"
Nov 03 18:07:42.828880 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:42.828852    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bcb2fe-aef1-446a-9e93-a18dc824d388-credential-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69\" (UID: \"a1bcb2fe-aef1-446a-9e93-a18dc824d388\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69"
Nov 03 18:07:42.833220 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:42.833189    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b48dcdb-9a6d-4e03-854f-ece61e03990e-maas-api-xks-ca" pod "6b48dcdb-9a6d-4e03-854f-ece61e03990e" (UID: "6b48dcdb-9a6d-4e03-854f-ece61e03990e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:07:42.833432 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:42.833304    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1bcb2fe-aef1-446a-9e93-a18dc824d388-istio-envoy\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69\" (UID: \"a1bcb2fe-aef1-446a-9e93-a18dc824d388\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69"
Nov 03 18:07:42.833432 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:42.833312    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b48dcdb-9a6d-4e03-854f-ece61e03990e-kube-api-access-vrwr2" pod "6b48dcdb-9a6d-4e03-854f-ece61e03990e" (UID: "6b48dcdb-9a6d-4e03-854f-ece61e03990e"). InnerVolumeSpecName "kube-api-access-vrwr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:07:42.833805 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:42.833776    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1bcb2fe-aef1-446a-9e93-a18dc824d388-istio-podinfo\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69\" (UID: \"a1bcb2fe-aef1-446a-9e93-a18dc824d388\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69"
Nov 03 18:07:42.886351 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:42.886317    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1bcb2fe-aef1-446a-9e93-a18dc824d388-istio-token\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69\" (UID: \"a1bcb2fe-aef1-446a-9e93-a18dc824d388\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69"
Nov 03 18:07:42.886351 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:42.886339    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c6r4\" (UniqueName: \"kubernetes.io/projected/a1bcb2fe-aef1-446a-9e93-a18dc824d388-kube-api-access-9c6r4\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69\" (UID: \"a1bcb2fe-aef1-446a-9e93-a18dc824d388\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69"
Nov 03 18:07:42.892186 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:42.892150    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69"
Nov 03 18:07:42.929205 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:42.929173    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b48dcdb-9a6d-4e03-854f-ece61e03990e-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:07:42.929205 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:42.929203    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrwr2\" (UniqueName: \"kubernetes.io/projected/6b48dcdb-9a6d-4e03-854f-ece61e03990e-kube-api-access-vrwr2\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:07:43.319045 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:43.319012    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b6e7259-854d546fff-j5qbr"
Nov 03 18:07:43.329720 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:43.329690    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69"]
Nov 03 18:07:43.333180 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:07:43.333152    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1bcb2fe_aef1_446a_9e93_a18dc824d388.slice/crio-f541bdcfdbc434a0b7e488ba8553e004de9a118ae409bf0b559818a521b95479 WatchSource:0}: Error finding container f541bdcfdbc434a0b7e488ba8553e004de9a118ae409bf0b559818a521b95479: Status 404 returned error can't find the container with id f541bdcfdbc434a0b7e488ba8553e004de9a118ae409bf0b559818a521b95479
Nov 03 18:07:43.335700 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:43.335664    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:07:43.335803 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:43.335748    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:07:43.335803 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:43.335795    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:07:43.495136 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:43.495093    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69" event={"ID":"a1bcb2fe-aef1-446a-9e93-a18dc824d388","Type":"ContainerStarted","Data":"0328b4a91e6c4a00505ad05fdbb246b65a375392a88f17f99f08423cef3b220c"}
Nov 03 18:07:43.495313 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:43.495196    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69" event={"ID":"a1bcb2fe-aef1-446a-9e93-a18dc824d388","Type":"ContainerStarted","Data":"f541bdcfdbc434a0b7e488ba8553e004de9a118ae409bf0b559818a521b95479"}
Nov 03 18:07:43.496796 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:43.496772    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-727fe8c5-gtqh9"
Nov 03 18:07:43.496912 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:43.496787    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-727fe8c5-gtqh9" event={"ID":"6b48dcdb-9a6d-4e03-854f-ece61e03990e","Type":"ContainerDied","Data":"5b71d5e1ea9ee7615f9823e6be0e7ccd38585e4422601d3213efb28cf1ede1ab"}
Nov 03 18:07:43.496912 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:43.496841    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b71d5e1ea9ee7615f9823e6be0e7ccd38585e4422601d3213efb28cf1ede1ab"
Nov 03 18:07:43.612551 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:43.612472    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69" podStartSLOduration=1.612454303 podStartE2EDuration="1.612454303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:07:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:07:43.608343781 +0000 UTC m=+1302.624541851" watchObservedRunningTime="2026-11-03 18:07:43.612454303 +0000 UTC m=+1302.628652372"
Nov 03 18:07:43.893091 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:43.893005    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69"
Nov 03 18:07:44.355186 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:44.355158    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-4b6e7259-54b9769cbc-k5qbr"
Nov 03 18:07:44.893910 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:44.893878    5315 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 03 18:07:44.894353 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:44.893945    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69" podUID="a1bcb2fe-aef1-446a-9e93-a18dc824d388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 03 18:07:45.892757 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:45.892720    5315 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 03 18:07:45.892948 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:45.892784    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69" podUID="a1bcb2fe-aef1-446a-9e93-a18dc824d388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 03 18:07:46.892799 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:46.892760    5315 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 03 18:07:46.893269 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:46.892851    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69" podUID="a1bcb2fe-aef1-446a-9e93-a18dc824d388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 03 18:07:47.892678 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:47.892642    5315 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 03 18:07:47.892858 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:47.892725    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69" podUID="a1bcb2fe-aef1-446a-9e93-a18dc824d388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 03 18:07:48.893541 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:48.893504    5315 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 03 18:07:48.893946 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:48.893563    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69" podUID="a1bcb2fe-aef1-446a-9e93-a18dc824d388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 03 18:07:49.899117 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:49.899080    5315 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 03 18:07:49.899758 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:49.899724    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69" podUID="a1bcb2fe-aef1-446a-9e93-a18dc824d388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 03 18:07:50.897431 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:50.897397    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69"
Nov 03 18:07:50.897974 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:50.897950    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69"
Nov 03 18:07:50.898554 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:07:50.898532    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69"
Nov 03 18:07:54.579486 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:07:54.579448    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:08:01.559635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:01.554206    5315 scope.go:122] "RemoveContainer" containerID="641315f6c7fd9f903a02c40ac1a7d47abba2e98d7db410274656840b234e909f"
Nov 03 18:08:01.702188 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:01.701762    5315 scope.go:122] "RemoveContainer" containerID="9db42ad22a2ceb767968f1d11b04743dfd2459a596ad9347b29b3183f51b3c5b"
Nov 03 18:08:01.907792 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:01.907660    5315 scope.go:122] "RemoveContainer" containerID="449c3097706febf9f9537d4eff7b6e383f8afbf56b014acb0502d2025c0e6f77"
Nov 03 18:08:02.005198 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:02.005167    5315 scope.go:122] "RemoveContainer" containerID="d0d1932ca54da64aee3d19de587b7082f2cb29e6520da6b61011d37222e164df"
Nov 03 18:08:02.075665 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:02.075517    5315 scope.go:122] "RemoveContainer" containerID="03e894e4d837336e8c020126a0448f44dba33db43979559a0977a4c59f2c3d95"
Nov 03 18:08:02.095880 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:02.095805    5315 scope.go:122] "RemoveContainer" containerID="c494d0b3b02155e3f21621c1c576ca19ed8f5748941568380adbc8c46d4504bc"
Nov 03 18:08:02.115773 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:02.115740    5315 scope.go:122] "RemoveContainer" containerID="73496c79e43f0cc691a8553671bfcbc8758b3b840c5ffa7666abc35c208f3c4d"
Nov 03 18:08:02.131843 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:02.128679    5315 scope.go:122] "RemoveContainer" containerID="5d1c57937f47cdedfa53466d62f62c47eb29eee45c234c50714044681513988e"
Nov 03 18:08:02.161191 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:02.161163    5315 scope.go:122] "RemoveContainer" containerID="63315cec3c5329265b683589e4aca34519005de46bdd6367aa762e410e5caec2"
Nov 03 18:08:02.171675 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:02.171649    5315 scope.go:122] "RemoveContainer" containerID="0fd60b94e70fe236fc7ad1f80a53943d9255e5885ab2f2e3e4ab09fb64e20c3d"
Nov 03 18:08:02.182226 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:02.182202    5315 scope.go:122] "RemoveContainer" containerID="18fe9b6516b7171208776b04259ec7b7f3aacafc6410d9e9849d6ea1b6df1d88"
Nov 03 18:08:10.129776 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.129730    5315 patch_prober.go:29] interesting pod/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:08:10.130177 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.129813    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns" podUID="8832b3d8-1880-4277-8a0c-f9bf94eef7b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:08:10.519103 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.519021    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f26ceb6a-4rd6s"]
Nov 03 18:08:10.519858 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.519840    5315 cpu_manager.go:414] "containerMap: removing container" podUID="6b48dcdb-9a6d-4e03-854f-ece61e03990e" containerName="revoke-keys"
Nov 03 18:08:10.519917 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.519860    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b48dcdb-9a6d-4e03-854f-ece61e03990e" containerName="revoke-keys"
Nov 03 18:08:10.519986 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.519975    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b48dcdb-9a6d-4e03-854f-ece61e03990e" containerName="revoke-keys"
Nov 03 18:08:10.522731 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.522710    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f26ceb6a-4rd6s"
Nov 03 18:08:10.575385 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.575346    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee361c55-2d7d-4b37-b1c5-3d4f29868fd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f26ceb6a-4rd6s\" (UID: \"ee361c55-2d7d-4b37-b1c5-3d4f29868fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f26ceb6a-4rd6s"
Nov 03 18:08:10.575566 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.575527    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee361c55-2d7d-4b37-b1c5-3d4f29868fd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f26ceb6a-4rd6s\" (UID: \"ee361c55-2d7d-4b37-b1c5-3d4f29868fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f26ceb6a-4rd6s"
Nov 03 18:08:10.575718 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.575700    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8smw9\" (UniqueName: \"kubernetes.io/projected/ee361c55-2d7d-4b37-b1c5-3d4f29868fd4-kube-api-access-8smw9\") pod \"maas-api-revoke-sub-f26ceb6a-4rd6s\" (UID: \"ee361c55-2d7d-4b37-b1c5-3d4f29868fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f26ceb6a-4rd6s"
Nov 03 18:08:10.617475 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.617443    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:08:10.617742 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.617444    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:08:10.617821 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.617495    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:08:10.617821 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.617523    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:08:10.622602 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.622576    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f26ceb6a-4rd6s"]
Nov 03 18:08:10.676848 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.676814    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee361c55-2d7d-4b37-b1c5-3d4f29868fd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f26ceb6a-4rd6s\" (UID: \"ee361c55-2d7d-4b37-b1c5-3d4f29868fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f26ceb6a-4rd6s"
Nov 03 18:08:10.677036 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.676911    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8smw9\" (UniqueName: \"kubernetes.io/projected/ee361c55-2d7d-4b37-b1c5-3d4f29868fd4-kube-api-access-8smw9\") pod \"maas-api-revoke-sub-f26ceb6a-4rd6s\" (UID: \"ee361c55-2d7d-4b37-b1c5-3d4f29868fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f26ceb6a-4rd6s"
Nov 03 18:08:10.677107 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.677044    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee361c55-2d7d-4b37-b1c5-3d4f29868fd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f26ceb6a-4rd6s\" (UID: \"ee361c55-2d7d-4b37-b1c5-3d4f29868fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f26ceb6a-4rd6s"
Nov 03 18:08:10.677779 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.677750    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee361c55-2d7d-4b37-b1c5-3d4f29868fd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f26ceb6a-4rd6s\" (UID: \"ee361c55-2d7d-4b37-b1c5-3d4f29868fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f26ceb6a-4rd6s"
Nov 03 18:08:10.681710 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.681681    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee361c55-2d7d-4b37-b1c5-3d4f29868fd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f26ceb6a-4rd6s\" (UID: \"ee361c55-2d7d-4b37-b1c5-3d4f29868fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f26ceb6a-4rd6s"
Nov 03 18:08:10.686473 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.686449    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg"]
Nov 03 18:08:10.690518 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.690500    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg"
Nov 03 18:08:10.705865 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.705835    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt"]
Nov 03 18:08:10.716754 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.716730    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt"
Nov 03 18:08:10.735267 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.735238    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8smw9\" (UniqueName: \"kubernetes.io/projected/ee361c55-2d7d-4b37-b1c5-3d4f29868fd4-kube-api-access-8smw9\") pod \"maas-api-revoke-sub-f26ceb6a-4rd6s\" (UID: \"ee361c55-2d7d-4b37-b1c5-3d4f29868fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f26ceb6a-4rd6s"
Nov 03 18:08:10.739125 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.739104    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-29bb4e29-openshift-default-dockercfg-c7vc8\""
Nov 03 18:08:10.754982 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.754948    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg"]
Nov 03 18:08:10.778665 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.778629    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-istio-podinfo\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg\" (UID: \"86e80b99-3e6f-41ae-8832-9cfbbf2e543a\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg"
Nov 03 18:08:10.778830 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.778690    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/feb037bd-264e-486b-aa7d-f3679b439981-workload-socket\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt\" (UID: \"feb037bd-264e-486b-aa7d-f3679b439981\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt"
Nov 03 18:08:10.778830 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.778720    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-istio-data\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg\" (UID: \"86e80b99-3e6f-41ae-8832-9cfbbf2e543a\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg"
Nov 03 18:08:10.778830 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.778753    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-workload-socket\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg\" (UID: \"86e80b99-3e6f-41ae-8832-9cfbbf2e543a\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg"
Nov 03 18:08:10.778830 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.778787    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-credential-socket\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg\" (UID: \"86e80b99-3e6f-41ae-8832-9cfbbf2e543a\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg"
Nov 03 18:08:10.779059 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.778849    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-workload-certs\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg\" (UID: \"86e80b99-3e6f-41ae-8832-9cfbbf2e543a\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg"
Nov 03 18:08:10.779059 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.778895    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96wb5\" (UniqueName: \"kubernetes.io/projected/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-kube-api-access-96wb5\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg\" (UID: \"86e80b99-3e6f-41ae-8832-9cfbbf2e543a\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg"
Nov 03 18:08:10.779059 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.778957    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/feb037bd-264e-486b-aa7d-f3679b439981-workload-certs\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt\" (UID: \"feb037bd-264e-486b-aa7d-f3679b439981\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt"
Nov 03 18:08:10.779059 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.778988    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-istio-envoy\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg\" (UID: \"86e80b99-3e6f-41ae-8832-9cfbbf2e543a\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg"
Nov 03 18:08:10.779059 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.779054    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/feb037bd-264e-486b-aa7d-f3679b439981-istio-token\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt\" (UID: \"feb037bd-264e-486b-aa7d-f3679b439981\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt"
Nov 03 18:08:10.779471 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.779226    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/feb037bd-264e-486b-aa7d-f3679b439981-credential-socket\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt\" (UID: \"feb037bd-264e-486b-aa7d-f3679b439981\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt"
Nov 03 18:08:10.779471 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.779308    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/feb037bd-264e-486b-aa7d-f3679b439981-istio-envoy\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt\" (UID: \"feb037bd-264e-486b-aa7d-f3679b439981\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt"
Nov 03 18:08:10.779471 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.779356    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/feb037bd-264e-486b-aa7d-f3679b439981-istiod-ca-cert\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt\" (UID: \"feb037bd-264e-486b-aa7d-f3679b439981\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt"
Nov 03 18:08:10.779471 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.779413    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/feb037bd-264e-486b-aa7d-f3679b439981-istio-podinfo\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt\" (UID: \"feb037bd-264e-486b-aa7d-f3679b439981\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt"
Nov 03 18:08:10.779471 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.779464    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-istio-token\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg\" (UID: \"86e80b99-3e6f-41ae-8832-9cfbbf2e543a\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg"
Nov 03 18:08:10.779750 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.779492    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-istiod-ca-cert\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg\" (UID: \"86e80b99-3e6f-41ae-8832-9cfbbf2e543a\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg"
Nov 03 18:08:10.779750 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.779528    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hww2q\" (UniqueName: \"kubernetes.io/projected/feb037bd-264e-486b-aa7d-f3679b439981-kube-api-access-hww2q\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt\" (UID: \"feb037bd-264e-486b-aa7d-f3679b439981\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt"
Nov 03 18:08:10.779750 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.779554    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/feb037bd-264e-486b-aa7d-f3679b439981-istio-data\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt\" (UID: \"feb037bd-264e-486b-aa7d-f3679b439981\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt"
Nov 03 18:08:10.810666 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.810541    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt"]
Nov 03 18:08:10.836197 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.836161    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f26ceb6a-4rd6s"
Nov 03 18:08:10.885415 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.881912    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-istio-envoy\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg\" (UID: \"86e80b99-3e6f-41ae-8832-9cfbbf2e543a\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg"
Nov 03 18:08:10.885415 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.881972    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/feb037bd-264e-486b-aa7d-f3679b439981-istio-token\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt\" (UID: \"feb037bd-264e-486b-aa7d-f3679b439981\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt"
Nov 03 18:08:10.885415 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.882063    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/feb037bd-264e-486b-aa7d-f3679b439981-credential-socket\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt\" (UID: \"feb037bd-264e-486b-aa7d-f3679b439981\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt"
Nov 03 18:08:10.885415 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.882122    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/feb037bd-264e-486b-aa7d-f3679b439981-istio-envoy\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt\" (UID: \"feb037bd-264e-486b-aa7d-f3679b439981\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt"
Nov 03 18:08:10.885415 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.882150    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/feb037bd-264e-486b-aa7d-f3679b439981-istiod-ca-cert\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt\" (UID: \"feb037bd-264e-486b-aa7d-f3679b439981\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt"
Nov 03 18:08:10.885415 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.882184    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/feb037bd-264e-486b-aa7d-f3679b439981-istio-podinfo\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt\" (UID: \"feb037bd-264e-486b-aa7d-f3679b439981\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt"
Nov 03 18:08:10.885415 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.882211    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-istio-token\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg\" (UID: \"86e80b99-3e6f-41ae-8832-9cfbbf2e543a\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg"
Nov 03 18:08:10.885415 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.882230    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-istiod-ca-cert\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg\" (UID: \"86e80b99-3e6f-41ae-8832-9cfbbf2e543a\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg"
Nov 03 18:08:10.885415 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.882264    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hww2q\" (UniqueName: \"kubernetes.io/projected/feb037bd-264e-486b-aa7d-f3679b439981-kube-api-access-hww2q\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt\" (UID: \"feb037bd-264e-486b-aa7d-f3679b439981\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt"
Nov 03 18:08:10.885415 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.882281    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/feb037bd-264e-486b-aa7d-f3679b439981-istio-data\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt\" (UID: \"feb037bd-264e-486b-aa7d-f3679b439981\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt"
Nov 03 18:08:10.885415 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.882331    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-istio-podinfo\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg\" (UID: \"86e80b99-3e6f-41ae-8832-9cfbbf2e543a\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg"
Nov 03 18:08:10.885415 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.882349    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/feb037bd-264e-486b-aa7d-f3679b439981-workload-socket\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt\" (UID: \"feb037bd-264e-486b-aa7d-f3679b439981\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt"
Nov 03 18:08:10.885415 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.882364    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-istio-data\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg\" (UID: \"86e80b99-3e6f-41ae-8832-9cfbbf2e543a\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg"
Nov 03 18:08:10.885415 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.882386    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-workload-socket\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg\" (UID: \"86e80b99-3e6f-41ae-8832-9cfbbf2e543a\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg"
Nov 03 18:08:10.885415 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.882409    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-credential-socket\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg\" (UID: \"86e80b99-3e6f-41ae-8832-9cfbbf2e543a\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg"
Nov 03 18:08:10.885415 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.882450    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-workload-certs\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg\" (UID: \"86e80b99-3e6f-41ae-8832-9cfbbf2e543a\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg"
Nov 03 18:08:10.885415 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.882480    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96wb5\" (UniqueName: \"kubernetes.io/projected/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-kube-api-access-96wb5\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg\" (UID: \"86e80b99-3e6f-41ae-8832-9cfbbf2e543a\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg"
Nov 03 18:08:10.885415 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.882526    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/feb037bd-264e-486b-aa7d-f3679b439981-workload-certs\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt\" (UID: \"feb037bd-264e-486b-aa7d-f3679b439981\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt"
Nov 03 18:08:10.885415 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.882882    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/feb037bd-264e-486b-aa7d-f3679b439981-workload-certs\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt\" (UID: \"feb037bd-264e-486b-aa7d-f3679b439981\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt"
Nov 03 18:08:10.886859 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.885834    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/feb037bd-264e-486b-aa7d-f3679b439981-credential-socket\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt\" (UID: \"feb037bd-264e-486b-aa7d-f3679b439981\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt"
Nov 03 18:08:10.886859 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.886084    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/feb037bd-264e-486b-aa7d-f3679b439981-istio-data\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt\" (UID: \"feb037bd-264e-486b-aa7d-f3679b439981\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt"
Nov 03 18:08:10.887077 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.887026    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-istio-envoy\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg\" (UID: \"86e80b99-3e6f-41ae-8832-9cfbbf2e543a\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg"
Nov 03 18:08:10.888267 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.888215    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-workload-socket\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg\" (UID: \"86e80b99-3e6f-41ae-8832-9cfbbf2e543a\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg"
Nov 03 18:08:10.888459 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.888434    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/feb037bd-264e-486b-aa7d-f3679b439981-istiod-ca-cert\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt\" (UID: \"feb037bd-264e-486b-aa7d-f3679b439981\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt"
Nov 03 18:08:10.888543 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.888464    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-istio-data\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg\" (UID: \"86e80b99-3e6f-41ae-8832-9cfbbf2e543a\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg"
Nov 03 18:08:10.888543 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.888477    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/feb037bd-264e-486b-aa7d-f3679b439981-workload-socket\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt\" (UID: \"feb037bd-264e-486b-aa7d-f3679b439981\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt"
Nov 03 18:08:10.888757 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.888735    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-workload-certs\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg\" (UID: \"86e80b99-3e6f-41ae-8832-9cfbbf2e543a\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg"
Nov 03 18:08:10.888947 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.888919    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-credential-socket\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg\" (UID: \"86e80b99-3e6f-41ae-8832-9cfbbf2e543a\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg"
Nov 03 18:08:10.889306 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.889289    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-istiod-ca-cert\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg\" (UID: \"86e80b99-3e6f-41ae-8832-9cfbbf2e543a\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg"
Nov 03 18:08:10.890228 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.890203    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/feb037bd-264e-486b-aa7d-f3679b439981-istio-envoy\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt\" (UID: \"feb037bd-264e-486b-aa7d-f3679b439981\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt"
Nov 03 18:08:10.891173 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.891141    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/feb037bd-264e-486b-aa7d-f3679b439981-istio-podinfo\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt\" (UID: \"feb037bd-264e-486b-aa7d-f3679b439981\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt"
Nov 03 18:08:10.892276 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.892249    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-istio-podinfo\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg\" (UID: \"86e80b99-3e6f-41ae-8832-9cfbbf2e543a\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg"
Nov 03 18:08:10.942944 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.942913    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/feb037bd-264e-486b-aa7d-f3679b439981-istio-token\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt\" (UID: \"feb037bd-264e-486b-aa7d-f3679b439981\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt"
Nov 03 18:08:10.943687 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.943566    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hww2q\" (UniqueName: \"kubernetes.io/projected/feb037bd-264e-486b-aa7d-f3679b439981-kube-api-access-hww2q\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt\" (UID: \"feb037bd-264e-486b-aa7d-f3679b439981\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt"
Nov 03 18:08:10.946101 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.946071    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-istio-token\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg\" (UID: \"86e80b99-3e6f-41ae-8832-9cfbbf2e543a\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg"
Nov 03 18:08:10.947502 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:10.947476    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96wb5\" (UniqueName: \"kubernetes.io/projected/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-kube-api-access-96wb5\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg\" (UID: \"86e80b99-3e6f-41ae-8832-9cfbbf2e543a\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg"
Nov 03 18:08:11.011529 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:11.011439    5315 generic.go:356] "Generic (PLEG): container finished" podID="8832b3d8-1880-4277-8a0c-f9bf94eef7b3" containerID="767d0ab15d11016a2d5140fad23874f7304cd7ecc9136f36983a08713a478858" exitCode=0
Nov 03 18:08:11.011714 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:11.011518    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns" event={"ID":"8832b3d8-1880-4277-8a0c-f9bf94eef7b3","Type":"ContainerDied","Data":"767d0ab15d11016a2d5140fad23874f7304cd7ecc9136f36983a08713a478858"}
Nov 03 18:08:11.012092 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:11.012058    5315 scope.go:122] "RemoveContainer" containerID="767d0ab15d11016a2d5140fad23874f7304cd7ecc9136f36983a08713a478858"
Nov 03 18:08:11.013762 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:11.013730    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:08:11.013852 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:11.013798    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:08:11.013852 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:11.013832    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:08:11.014380 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:11.014263    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg"
Nov 03 18:08:11.029543 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:11.029480    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt"
Nov 03 18:08:11.237504 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:11.237452    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f26ceb6a-4rd6s"]
Nov 03 18:08:11.416523 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:11.416497    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg"]
Nov 03 18:08:11.418872 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:08:11.418843    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86e80b99_3e6f_41ae_8832_9cfbbf2e543a.slice/crio-654b5fdf6fe85e3ddf58b6c3f2eaf7b2b7a6fa50235006ca55c7af9abc834097 WatchSource:0}: Error finding container 654b5fdf6fe85e3ddf58b6c3f2eaf7b2b7a6fa50235006ca55c7af9abc834097: Status 404 returned error can't find the container with id 654b5fdf6fe85e3ddf58b6c3f2eaf7b2b7a6fa50235006ca55c7af9abc834097
Nov 03 18:08:11.422291 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:11.422254    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:08:11.422422 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:11.422343    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:08:11.422422 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:11.422386    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:08:11.478929 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:11.471557    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt"]
Nov 03 18:08:11.486976 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:11.486940    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:08:11.487088 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:11.487020    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:08:11.487088 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:11.487065    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:08:12.066591 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:12.066365    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns" event={"ID":"8832b3d8-1880-4277-8a0c-f9bf94eef7b3","Type":"ContainerStarted","Data":"8b456c26e6f2ebf28ceb6430fbb16922ede514eda2e46a44d11cd169622edf3c"}
Nov 03 18:08:12.069677 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:12.069467    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f26ceb6a-4rd6s" event={"ID":"ee361c55-2d7d-4b37-b1c5-3d4f29868fd4","Type":"ContainerStarted","Data":"845ad1e41d79011c112e61c1577535866fb7a3612fa3375e2a4875e9d30e65fb"}
Nov 03 18:08:12.069677 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:12.069522    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f26ceb6a-4rd6s" event={"ID":"ee361c55-2d7d-4b37-b1c5-3d4f29868fd4","Type":"ContainerStarted","Data":"d515cb8799b819d2ecbf182cd1d2eea9bf59c7b4a36b57a777899e99dec5d094"}
Nov 03 18:08:12.072380 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:12.071573    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt" event={"ID":"feb037bd-264e-486b-aa7d-f3679b439981","Type":"ContainerStarted","Data":"51097ba4de28cb7a57daccd00acaff578898ba5cf1f9164d51ed3177999e2ed7"}
Nov 03 18:08:12.072380 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:12.071674    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt" event={"ID":"feb037bd-264e-486b-aa7d-f3679b439981","Type":"ContainerStarted","Data":"53dcaba89d5a00dd436e66a353d87e176b19c1d0e23afcfceec680e7f6e0ff7d"}
Nov 03 18:08:12.074299 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:12.074265    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg" event={"ID":"86e80b99-3e6f-41ae-8832-9cfbbf2e543a","Type":"ContainerStarted","Data":"0dfdd1466f5ab8b88a3e96c4d98823ffdbe87503efd084a453c49b0d9a80e2cf"}
Nov 03 18:08:12.074410 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:12.074324    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg" event={"ID":"86e80b99-3e6f-41ae-8832-9cfbbf2e543a","Type":"ContainerStarted","Data":"654b5fdf6fe85e3ddf58b6c3f2eaf7b2b7a6fa50235006ca55c7af9abc834097"}
Nov 03 18:08:12.151633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:12.151039    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns"
Nov 03 18:08:12.151633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:12.151175    5315 patch_prober.go:29] interesting pod/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Nov 03 18:08:12.151633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:12.151239    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns" podUID="8832b3d8-1880-4277-8a0c-f9bf94eef7b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Nov 03 18:08:12.386670 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:12.368382    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg" podStartSLOduration=2.368359989 podStartE2EDuration="2.368359989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:08:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:08:12.365017392 +0000 UTC m=+1331.381215462" watchObservedRunningTime="2026-11-03 18:08:12.368359989 +0000 UTC m=+1331.384558058"
Nov 03 18:08:12.629688 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:12.629415    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt" podStartSLOduration=2.629394801 podStartE2EDuration="2.629394801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:08:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:08:12.512414685 +0000 UTC m=+1331.528612756" watchObservedRunningTime="2026-11-03 18:08:12.629394801 +0000 UTC m=+1331.645592873"
Nov 03 18:08:12.694636 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:12.678601    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f26ceb6a-4rd6s" podStartSLOduration=2.67858155 podStartE2EDuration="2.67858155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:08:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:08:12.677861196 +0000 UTC m=+1331.694059266" watchObservedRunningTime="2026-11-03 18:08:12.67858155 +0000 UTC m=+1331.694779620"
Nov 03 18:08:13.066651 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:13.057817    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg"
Nov 03 18:08:13.066651 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:13.059681    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt"
Nov 03 18:08:13.067781 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:13.067740    5315 patch_prober.go:29] interesting pod/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 03 18:08:13.090502 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:13.067970    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt" podUID="feb037bd-264e-486b-aa7d-f3679b439981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 03 18:08:13.114100 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:13.109727    5315 patch_prober.go:29] interesting pod/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Nov 03 18:08:13.114100 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:13.109788    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg" podUID="86e80b99-3e6f-41ae-8832-9cfbbf2e543a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Nov 03 18:08:13.156634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:13.155760    5315 patch_prober.go:29] interesting pod/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Nov 03 18:08:13.156634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:13.155826    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns" podUID="8832b3d8-1880-4277-8a0c-f9bf94eef7b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Nov 03 18:08:13.178275 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:13.166170    5315 generic.go:356] "Generic (PLEG): container finished" podID="ee361c55-2d7d-4b37-b1c5-3d4f29868fd4" containerID="845ad1e41d79011c112e61c1577535866fb7a3612fa3375e2a4875e9d30e65fb" exitCode=0
Nov 03 18:08:13.178275 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:13.170775    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f26ceb6a-4rd6s" event={"ID":"ee361c55-2d7d-4b37-b1c5-3d4f29868fd4","Type":"ContainerDied","Data":"845ad1e41d79011c112e61c1577535866fb7a3612fa3375e2a4875e9d30e65fb"}
Nov 03 18:08:14.014932 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:14.014889    5315 patch_prober.go:29] interesting pod/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Nov 03 18:08:14.015440 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:14.014959    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg" podUID="86e80b99-3e6f-41ae-8832-9cfbbf2e543a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Nov 03 18:08:14.039633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:14.030012    5315 patch_prober.go:29] interesting pod/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 03 18:08:14.039633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:14.030074    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt" podUID="feb037bd-264e-486b-aa7d-f3679b439981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 03 18:08:14.151633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:14.151135    5315 patch_prober.go:29] interesting pod/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Nov 03 18:08:14.151633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:14.151200    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns" podUID="8832b3d8-1880-4277-8a0c-f9bf94eef7b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Nov 03 18:08:14.867890 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:14.867847    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f26ceb6a-4rd6s"
Nov 03 18:08:14.934692 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:14.934135    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee361c55-2d7d-4b37-b1c5-3d4f29868fd4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee361c55-2d7d-4b37-b1c5-3d4f29868fd4-maas-api-xks-ca\") pod \"ee361c55-2d7d-4b37-b1c5-3d4f29868fd4\" (UID: \"ee361c55-2d7d-4b37-b1c5-3d4f29868fd4\") "
Nov 03 18:08:14.934692 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:14.934237    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee361c55-2d7d-4b37-b1c5-3d4f29868fd4-kube-api-access-8smw9\" (UniqueName: \"kubernetes.io/projected/ee361c55-2d7d-4b37-b1c5-3d4f29868fd4-kube-api-access-8smw9\") pod \"ee361c55-2d7d-4b37-b1c5-3d4f29868fd4\" (UID: \"ee361c55-2d7d-4b37-b1c5-3d4f29868fd4\") "
Nov 03 18:08:14.934692 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:14.934462    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee361c55-2d7d-4b37-b1c5-3d4f29868fd4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee361c55-2d7d-4b37-b1c5-3d4f29868fd4-maas-api-openshift-ca\") pod \"ee361c55-2d7d-4b37-b1c5-3d4f29868fd4\" (UID: \"ee361c55-2d7d-4b37-b1c5-3d4f29868fd4\") "
Nov 03 18:08:14.954633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:14.951821    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee361c55-2d7d-4b37-b1c5-3d4f29868fd4-maas-api-openshift-ca" pod "ee361c55-2d7d-4b37-b1c5-3d4f29868fd4" (UID: "ee361c55-2d7d-4b37-b1c5-3d4f29868fd4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:08:14.977889 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:14.976156    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee361c55-2d7d-4b37-b1c5-3d4f29868fd4-maas-api-xks-ca" pod "ee361c55-2d7d-4b37-b1c5-3d4f29868fd4" (UID: "ee361c55-2d7d-4b37-b1c5-3d4f29868fd4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:08:15.001877 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:15.001764    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee361c55-2d7d-4b37-b1c5-3d4f29868fd4-kube-api-access-8smw9" pod "ee361c55-2d7d-4b37-b1c5-3d4f29868fd4" (UID: "ee361c55-2d7d-4b37-b1c5-3d4f29868fd4"). InnerVolumeSpecName "kube-api-access-8smw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:08:15.015203 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:15.015166    5315 patch_prober.go:29] interesting pod/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Nov 03 18:08:15.015731 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:15.015224    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg" podUID="86e80b99-3e6f-41ae-8832-9cfbbf2e543a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Nov 03 18:08:15.032650 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:15.031025    5315 patch_prober.go:29] interesting pod/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 03 18:08:15.032650 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:15.031087    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt" podUID="feb037bd-264e-486b-aa7d-f3679b439981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 03 18:08:15.052668 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:15.037023    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee361c55-2d7d-4b37-b1c5-3d4f29868fd4-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:08:15.052668 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:15.037063    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee361c55-2d7d-4b37-b1c5-3d4f29868fd4-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:08:15.052668 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:15.037080    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8smw9\" (UniqueName: \"kubernetes.io/projected/ee361c55-2d7d-4b37-b1c5-3d4f29868fd4-kube-api-access-8smw9\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:08:15.186694 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:15.182430    5315 patch_prober.go:29] interesting pod/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Nov 03 18:08:15.186694 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:15.182497    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns" podUID="8832b3d8-1880-4277-8a0c-f9bf94eef7b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Nov 03 18:08:15.268411 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:15.223665    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f26ceb6a-4rd6s"
Nov 03 18:08:15.268411 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:15.223678    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f26ceb6a-4rd6s" event={"ID":"ee361c55-2d7d-4b37-b1c5-3d4f29868fd4","Type":"ContainerDied","Data":"d515cb8799b819d2ecbf182cd1d2eea9bf59c7b4a36b57a777899e99dec5d094"}
Nov 03 18:08:15.268411 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:15.223737    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d515cb8799b819d2ecbf182cd1d2eea9bf59c7b4a36b57a777899e99dec5d094"
Nov 03 18:08:16.015556 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:16.015519    5315 patch_prober.go:29] interesting pod/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Nov 03 18:08:16.015979 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:16.015574    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg" podUID="86e80b99-3e6f-41ae-8832-9cfbbf2e543a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Nov 03 18:08:16.031633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:16.029966    5315 patch_prober.go:29] interesting pod/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 03 18:08:16.031633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:16.030034    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt" podUID="feb037bd-264e-486b-aa7d-f3679b439981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 03 18:08:16.163634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:16.163070    5315 patch_prober.go:29] interesting pod/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Nov 03 18:08:16.163634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:16.163136    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns" podUID="8832b3d8-1880-4277-8a0c-f9bf94eef7b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Nov 03 18:08:17.014873 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:17.014829    5315 patch_prober.go:29] interesting pod/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Nov 03 18:08:17.015079 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:17.014890    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg" podUID="86e80b99-3e6f-41ae-8832-9cfbbf2e543a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Nov 03 18:08:17.029989 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:17.029950    5315 patch_prober.go:29] interesting pod/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 03 18:08:17.030654 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:17.030590    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt" podUID="feb037bd-264e-486b-aa7d-f3679b439981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 03 18:08:17.154121 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:17.154078    5315 patch_prober.go:29] interesting pod/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Nov 03 18:08:17.154406 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:17.154372    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns" podUID="8832b3d8-1880-4277-8a0c-f9bf94eef7b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Nov 03 18:08:18.014777 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:18.014735    5315 patch_prober.go:29] interesting pod/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Nov 03 18:08:18.014969 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:18.014793    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg" podUID="86e80b99-3e6f-41ae-8832-9cfbbf2e543a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Nov 03 18:08:18.038049 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:18.038008    5315 patch_prober.go:29] interesting pod/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 03 18:08:18.038658 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:18.038603    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt" podUID="feb037bd-264e-486b-aa7d-f3679b439981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 03 18:08:18.153565 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:18.153531    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns"
Nov 03 18:08:18.154132 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:18.154116    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns"
Nov 03 18:08:18.155025 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:18.155002    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns"
Nov 03 18:08:19.019012 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:19.018979    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg"
Nov 03 18:08:19.019511 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:19.019485    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg"
Nov 03 18:08:19.020122 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:19.020100    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg"
Nov 03 18:08:19.033966 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:19.033943    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt"
Nov 03 18:08:19.034345 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:19.034323    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt"
Nov 03 18:08:19.035085 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:19.035066    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt"
Nov 03 18:08:22.964112 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:22.964077    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-86101f5d-6ffcj"]
Nov 03 18:08:22.964844 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:22.964828    5315 cpu_manager.go:414] "containerMap: removing container" podUID="ee361c55-2d7d-4b37-b1c5-3d4f29868fd4" containerName="revoke-keys"
Nov 03 18:08:22.964844 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:22.964845    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee361c55-2d7d-4b37-b1c5-3d4f29868fd4" containerName="revoke-keys"
Nov 03 18:08:22.965015 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:22.965002    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee361c55-2d7d-4b37-b1c5-3d4f29868fd4" containerName="revoke-keys"
Nov 03 18:08:22.968917 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:22.968899    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-86101f5d-6ffcj"
Nov 03 18:08:23.018083 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:23.018046    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:08:23.063858 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:23.063822    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:08:23.063858 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:23.063823    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:08:23.064062 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:23.063963    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:08:23.065862 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:23.065839    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b64f8fe2-0b09-47d9-981c-3dd32f24169f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-86101f5d-6ffcj\" (UID: \"b64f8fe2-0b09-47d9-981c-3dd32f24169f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-86101f5d-6ffcj"
Nov 03 18:08:23.066097 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:23.066074    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b64f8fe2-0b09-47d9-981c-3dd32f24169f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-86101f5d-6ffcj\" (UID: \"b64f8fe2-0b09-47d9-981c-3dd32f24169f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-86101f5d-6ffcj"
Nov 03 18:08:23.066173 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:23.066143    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg954\" (UniqueName: \"kubernetes.io/projected/b64f8fe2-0b09-47d9-981c-3dd32f24169f-kube-api-access-pg954\") pod \"maas-api-revoke-keys-e2e-migrate-86101f5d-6ffcj\" (UID: \"b64f8fe2-0b09-47d9-981c-3dd32f24169f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-86101f5d-6ffcj"
Nov 03 18:08:23.069745 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:23.069722    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-86101f5d-6ffcj"]
Nov 03 18:08:23.167211 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:23.167178    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg954\" (UniqueName: \"kubernetes.io/projected/b64f8fe2-0b09-47d9-981c-3dd32f24169f-kube-api-access-pg954\") pod \"maas-api-revoke-keys-e2e-migrate-86101f5d-6ffcj\" (UID: \"b64f8fe2-0b09-47d9-981c-3dd32f24169f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-86101f5d-6ffcj"
Nov 03 18:08:23.167382 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:23.167344    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b64f8fe2-0b09-47d9-981c-3dd32f24169f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-86101f5d-6ffcj\" (UID: \"b64f8fe2-0b09-47d9-981c-3dd32f24169f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-86101f5d-6ffcj"
Nov 03 18:08:23.167538 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:23.167518    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b64f8fe2-0b09-47d9-981c-3dd32f24169f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-86101f5d-6ffcj\" (UID: \"b64f8fe2-0b09-47d9-981c-3dd32f24169f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-86101f5d-6ffcj"
Nov 03 18:08:23.167984 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:23.167964    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b64f8fe2-0b09-47d9-981c-3dd32f24169f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-86101f5d-6ffcj\" (UID: \"b64f8fe2-0b09-47d9-981c-3dd32f24169f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-86101f5d-6ffcj"
Nov 03 18:08:23.169778 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:23.169758    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b64f8fe2-0b09-47d9-981c-3dd32f24169f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-86101f5d-6ffcj\" (UID: \"b64f8fe2-0b09-47d9-981c-3dd32f24169f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-86101f5d-6ffcj"
Nov 03 18:08:23.224796 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:23.224726    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg954\" (UniqueName: \"kubernetes.io/projected/b64f8fe2-0b09-47d9-981c-3dd32f24169f-kube-api-access-pg954\") pod \"maas-api-revoke-keys-e2e-migrate-86101f5d-6ffcj\" (UID: \"b64f8fe2-0b09-47d9-981c-3dd32f24169f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-86101f5d-6ffcj"
Nov 03 18:08:23.283767 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:23.283738    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-86101f5d-6ffcj"
Nov 03 18:08:23.662792 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:23.662759    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-86101f5d-6ffcj"]
Nov 03 18:08:23.667406 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:08:23.667379    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb64f8fe2_0b09_47d9_981c_3dd32f24169f.slice/crio-4475e90d00c2a5330fac021233f4c1013bd403b9210a0ee741a1dd75e3ec22a4 WatchSource:0}: Error finding container 4475e90d00c2a5330fac021233f4c1013bd403b9210a0ee741a1dd75e3ec22a4: Status 404 returned error can't find the container with id 4475e90d00c2a5330fac021233f4c1013bd403b9210a0ee741a1dd75e3ec22a4
Nov 03 18:08:24.429675 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:24.429637    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-86101f5d-6ffcj" event={"ID":"b64f8fe2-0b09-47d9-981c-3dd32f24169f","Type":"ContainerStarted","Data":"581dfc7b11bfd350ef5d7bf45ae44da894484af675859a287adaa3cf2ca10a8f"}
Nov 03 18:08:24.430129 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:24.429709    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-86101f5d-6ffcj" event={"ID":"b64f8fe2-0b09-47d9-981c-3dd32f24169f","Type":"ContainerStarted","Data":"4475e90d00c2a5330fac021233f4c1013bd403b9210a0ee741a1dd75e3ec22a4"}
Nov 03 18:08:24.540500 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:24.540446    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-86101f5d-6ffcj" podStartSLOduration=2.540432264 podStartE2EDuration="2.540432264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:08:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:08:24.537871653 +0000 UTC m=+1343.554069734" watchObservedRunningTime="2026-11-03 18:08:24.540432264 +0000 UTC m=+1343.556630332"
Nov 03 18:08:24.579841 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:08:24.579799    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:08:25.541629 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:25.540724    5315 generic.go:356] "Generic (PLEG): container finished" podID="b64f8fe2-0b09-47d9-981c-3dd32f24169f" containerID="581dfc7b11bfd350ef5d7bf45ae44da894484af675859a287adaa3cf2ca10a8f" exitCode=0
Nov 03 18:08:25.541629 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:25.540907    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-86101f5d-6ffcj" event={"ID":"b64f8fe2-0b09-47d9-981c-3dd32f24169f","Type":"ContainerDied","Data":"581dfc7b11bfd350ef5d7bf45ae44da894484af675859a287adaa3cf2ca10a8f"}
Nov 03 18:08:27.480894 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:27.480805    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-86101f5d-6ffcj"
Nov 03 18:08:27.533245 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:27.533205    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b64f8fe2-0b09-47d9-981c-3dd32f24169f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b64f8fe2-0b09-47d9-981c-3dd32f24169f-maas-api-xks-ca\") pod \"b64f8fe2-0b09-47d9-981c-3dd32f24169f\" (UID: \"b64f8fe2-0b09-47d9-981c-3dd32f24169f\") "
Nov 03 18:08:27.533830 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:27.533805    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b64f8fe2-0b09-47d9-981c-3dd32f24169f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b64f8fe2-0b09-47d9-981c-3dd32f24169f-maas-api-openshift-ca\") pod \"b64f8fe2-0b09-47d9-981c-3dd32f24169f\" (UID: \"b64f8fe2-0b09-47d9-981c-3dd32f24169f\") "
Nov 03 18:08:27.539632 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:27.533954    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b64f8fe2-0b09-47d9-981c-3dd32f24169f-kube-api-access-pg954\" (UniqueName: \"kubernetes.io/projected/b64f8fe2-0b09-47d9-981c-3dd32f24169f-kube-api-access-pg954\") pod \"b64f8fe2-0b09-47d9-981c-3dd32f24169f\" (UID: \"b64f8fe2-0b09-47d9-981c-3dd32f24169f\") "
Nov 03 18:08:27.548091 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:27.548057    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b64f8fe2-0b09-47d9-981c-3dd32f24169f-maas-api-xks-ca" pod "b64f8fe2-0b09-47d9-981c-3dd32f24169f" (UID: "b64f8fe2-0b09-47d9-981c-3dd32f24169f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:08:27.551796 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:27.551768    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b64f8fe2-0b09-47d9-981c-3dd32f24169f-maas-api-openshift-ca" pod "b64f8fe2-0b09-47d9-981c-3dd32f24169f" (UID: "b64f8fe2-0b09-47d9-981c-3dd32f24169f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:08:27.610379 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:27.610342    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b64f8fe2-0b09-47d9-981c-3dd32f24169f-kube-api-access-pg954" pod "b64f8fe2-0b09-47d9-981c-3dd32f24169f" (UID: "b64f8fe2-0b09-47d9-981c-3dd32f24169f"). InnerVolumeSpecName "kube-api-access-pg954". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:08:27.651633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:27.651118    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b64f8fe2-0b09-47d9-981c-3dd32f24169f-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:08:27.651633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:27.651165    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg954\" (UniqueName: \"kubernetes.io/projected/b64f8fe2-0b09-47d9-981c-3dd32f24169f-kube-api-access-pg954\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:08:27.651633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:27.651182    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b64f8fe2-0b09-47d9-981c-3dd32f24169f-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:08:27.677094 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:27.677049    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-86101f5d-6ffcj" event={"ID":"b64f8fe2-0b09-47d9-981c-3dd32f24169f","Type":"ContainerDied","Data":"4475e90d00c2a5330fac021233f4c1013bd403b9210a0ee741a1dd75e3ec22a4"}
Nov 03 18:08:27.677357 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:27.677339    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4475e90d00c2a5330fac021233f4c1013bd403b9210a0ee741a1dd75e3ec22a4"
Nov 03 18:08:27.677530 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:27.677518    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-86101f5d-6ffcj"
Nov 03 18:08:28.585635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:28.585138    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr" podUID="08159a02-fc2e-4138-824d-be5ba3e141d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:08:28.640827 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:28.640782    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s" podUID="155e4b39-1877-4bd9-8541-5d6bd3aa0a8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:08:33.637620 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:33.637567    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-86101f5d-78fb6dfcd7-4hcbt"]
Nov 03 18:08:37.137544 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:37.137504    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-29bb4e29-42cgh"]
Nov 03 18:08:37.138312 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:37.138294    5315 cpu_manager.go:414] "containerMap: removing container" podUID="b64f8fe2-0b09-47d9-981c-3dd32f24169f" containerName="revoke-keys"
Nov 03 18:08:37.138312 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:37.138313    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b64f8fe2-0b09-47d9-981c-3dd32f24169f" containerName="revoke-keys"
Nov 03 18:08:37.138428 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:37.138419    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="b64f8fe2-0b09-47d9-981c-3dd32f24169f" containerName="revoke-keys"
Nov 03 18:08:37.175555 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:37.175528    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-29bb4e29-42cgh"
Nov 03 18:08:37.194714 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:37.194676    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-29bb4e29-42cgh"]
Nov 03 18:08:37.231667 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:37.231589    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:08:37.231667 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:37.231597    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:08:37.231904 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:37.231695    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:08:37.270912 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:37.270881    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:08:37.288938 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:37.288901    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d05f048d-1346-491b-bc11-121a6c7d53a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-29bb4e29-42cgh\" (UID: \"d05f048d-1346-491b-bc11-121a6c7d53a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-29bb4e29-42cgh"
Nov 03 18:08:37.289103 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:37.289021    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpg8j\" (UniqueName: \"kubernetes.io/projected/d05f048d-1346-491b-bc11-121a6c7d53a6-kube-api-access-zpg8j\") pod \"maas-api-revoke-keys-e2e-ait-29bb4e29-42cgh\" (UID: \"d05f048d-1346-491b-bc11-121a6c7d53a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-29bb4e29-42cgh"
Nov 03 18:08:37.289237 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:37.289219    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d05f048d-1346-491b-bc11-121a6c7d53a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-29bb4e29-42cgh\" (UID: \"d05f048d-1346-491b-bc11-121a6c7d53a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-29bb4e29-42cgh"
Nov 03 18:08:37.390511 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:37.390422    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpg8j\" (UniqueName: \"kubernetes.io/projected/d05f048d-1346-491b-bc11-121a6c7d53a6-kube-api-access-zpg8j\") pod \"maas-api-revoke-keys-e2e-ait-29bb4e29-42cgh\" (UID: \"d05f048d-1346-491b-bc11-121a6c7d53a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-29bb4e29-42cgh"
Nov 03 18:08:37.390760 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:37.390581    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d05f048d-1346-491b-bc11-121a6c7d53a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-29bb4e29-42cgh\" (UID: \"d05f048d-1346-491b-bc11-121a6c7d53a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-29bb4e29-42cgh"
Nov 03 18:08:37.390760 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:37.390715    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d05f048d-1346-491b-bc11-121a6c7d53a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-29bb4e29-42cgh\" (UID: \"d05f048d-1346-491b-bc11-121a6c7d53a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-29bb4e29-42cgh"
Nov 03 18:08:37.391359 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:37.391333    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d05f048d-1346-491b-bc11-121a6c7d53a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-29bb4e29-42cgh\" (UID: \"d05f048d-1346-491b-bc11-121a6c7d53a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-29bb4e29-42cgh"
Nov 03 18:08:37.393058 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:37.393028    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d05f048d-1346-491b-bc11-121a6c7d53a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-29bb4e29-42cgh\" (UID: \"d05f048d-1346-491b-bc11-121a6c7d53a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-29bb4e29-42cgh"
Nov 03 18:08:37.448165 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:37.448134    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpg8j\" (UniqueName: \"kubernetes.io/projected/d05f048d-1346-491b-bc11-121a6c7d53a6-kube-api-access-zpg8j\") pod \"maas-api-revoke-keys-e2e-ait-29bb4e29-42cgh\" (UID: \"d05f048d-1346-491b-bc11-121a6c7d53a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-29bb4e29-42cgh"
Nov 03 18:08:37.492221 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:37.492188    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-29bb4e29-42cgh"
Nov 03 18:08:37.977882 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:37.977799    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-29bb4e29-42cgh"]
Nov 03 18:08:37.981737 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:08:37.981704    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd05f048d_1346_491b_bc11_121a6c7d53a6.slice/crio-99118c3dfc18aab2a5ee4a4354ef7fc858feddc9b8732e24854aebfbd79d1de8 WatchSource:0}: Error finding container 99118c3dfc18aab2a5ee4a4354ef7fc858feddc9b8732e24854aebfbd79d1de8: Status 404 returned error can't find the container with id 99118c3dfc18aab2a5ee4a4354ef7fc858feddc9b8732e24854aebfbd79d1de8
Nov 03 18:08:38.878832 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:38.878783    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-86101f5d-78fb6dfcd7-4hcbt" podUID="e3b1770a-983e-4a1f-8476-c37d1f5fe492" containerName="payload-pre-processing" containerID="cri-o://960f0a4c0b29d0fc75234db263642b653380ab56a05a7739e30e42e48c99a083" gracePeriod=25
Nov 03 18:08:38.926020 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:38.925983    5315 generic.go:356] "Generic (PLEG): container finished" podID="11671eb9-4824-4e89-9692-b5be9038c549" containerID="31303af85f59ce20708ef7dfcdeebc3fe83cf483a50ab05bf91b7020b28cb973" exitCode=0
Nov 03 18:08:38.926150 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:38.926054    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q" event={"ID":"11671eb9-4824-4e89-9692-b5be9038c549","Type":"ContainerDied","Data":"31303af85f59ce20708ef7dfcdeebc3fe83cf483a50ab05bf91b7020b28cb973"}
Nov 03 18:08:38.926584 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:38.926561    5315 scope.go:122] "RemoveContainer" containerID="31303af85f59ce20708ef7dfcdeebc3fe83cf483a50ab05bf91b7020b28cb973"
Nov 03 18:08:38.927960 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:38.927929    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-29bb4e29-42cgh" event={"ID":"d05f048d-1346-491b-bc11-121a6c7d53a6","Type":"ContainerStarted","Data":"0313a23b322ac08662841f6212b191959e8905ecde57612eca1e2535334072f0"}
Nov 03 18:08:38.928070 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:38.927988    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-29bb4e29-42cgh" event={"ID":"d05f048d-1346-491b-bc11-121a6c7d53a6","Type":"ContainerStarted","Data":"99118c3dfc18aab2a5ee4a4354ef7fc858feddc9b8732e24854aebfbd79d1de8"}
Nov 03 18:08:38.928321 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:38.928292    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:08:38.928375 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:38.928362    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:08:38.928414 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:38.928397    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:08:39.158535 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:39.158509    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-86101f5d-78fb6dfcd7-4hcbt"
Nov 03 18:08:39.234469 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:39.234411    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-29bb4e29-42cgh" podStartSLOduration=2.23439675 podStartE2EDuration="2.23439675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:08:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:08:39.232240534 +0000 UTC m=+1358.248438604" watchObservedRunningTime="2026-11-03 18:08:39.23439675 +0000 UTC m=+1358.250594818"
Nov 03 18:08:39.241286 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:39.241248    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3b1770a-983e-4a1f-8476-c37d1f5fe492-kube-api-access-g5tht\" (UniqueName: \"kubernetes.io/projected/e3b1770a-983e-4a1f-8476-c37d1f5fe492-kube-api-access-g5tht\") pod \"e3b1770a-983e-4a1f-8476-c37d1f5fe492\" (UID: \"e3b1770a-983e-4a1f-8476-c37d1f5fe492\") "
Nov 03 18:08:39.241464 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:39.241414    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3b1770a-983e-4a1f-8476-c37d1f5fe492-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3b1770a-983e-4a1f-8476-c37d1f5fe492-plugins-config-volume\") pod \"e3b1770a-983e-4a1f-8476-c37d1f5fe492\" (UID: \"e3b1770a-983e-4a1f-8476-c37d1f5fe492\") "
Nov 03 18:08:39.241837 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:39.241812    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3b1770a-983e-4a1f-8476-c37d1f5fe492-plugins-config-volume" pod "e3b1770a-983e-4a1f-8476-c37d1f5fe492" (UID: "e3b1770a-983e-4a1f-8476-c37d1f5fe492"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:08:39.242901 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:39.242771    5315 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3b1770a-983e-4a1f-8476-c37d1f5fe492-plugins-config-volume\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:08:39.243456 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:39.243432    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3b1770a-983e-4a1f-8476-c37d1f5fe492-kube-api-access-g5tht" pod "e3b1770a-983e-4a1f-8476-c37d1f5fe492" (UID: "e3b1770a-983e-4a1f-8476-c37d1f5fe492"). InnerVolumeSpecName "kube-api-access-g5tht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:08:39.344474 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:39.344435    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5tht\" (UniqueName: \"kubernetes.io/projected/e3b1770a-983e-4a1f-8476-c37d1f5fe492-kube-api-access-g5tht\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:08:39.962141 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:39.962097    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q" event={"ID":"11671eb9-4824-4e89-9692-b5be9038c549","Type":"ContainerStarted","Data":"12f02d737977cbccd0d09199b884317ef35871693aca2a93336aefa5081dfdb3"}
Nov 03 18:08:39.967114 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:39.967081    5315 generic.go:356] "Generic (PLEG): container finished" podID="d05f048d-1346-491b-bc11-121a6c7d53a6" containerID="0313a23b322ac08662841f6212b191959e8905ecde57612eca1e2535334072f0" exitCode=0
Nov 03 18:08:39.967273 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:39.967185    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-29bb4e29-42cgh" event={"ID":"d05f048d-1346-491b-bc11-121a6c7d53a6","Type":"ContainerDied","Data":"0313a23b322ac08662841f6212b191959e8905ecde57612eca1e2535334072f0"}
Nov 03 18:08:39.968733 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:39.968706    5315 generic.go:356] "Generic (PLEG): container finished" podID="e3b1770a-983e-4a1f-8476-c37d1f5fe492" containerID="960f0a4c0b29d0fc75234db263642b653380ab56a05a7739e30e42e48c99a083" exitCode=0
Nov 03 18:08:39.968843 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:39.968751    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-86101f5d-78fb6dfcd7-4hcbt"
Nov 03 18:08:39.968843 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:39.968784    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-86101f5d-78fb6dfcd7-4hcbt" event={"ID":"e3b1770a-983e-4a1f-8476-c37d1f5fe492","Type":"ContainerDied","Data":"960f0a4c0b29d0fc75234db263642b653380ab56a05a7739e30e42e48c99a083"}
Nov 03 18:08:39.968843 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:39.968833    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-86101f5d-78fb6dfcd7-4hcbt" event={"ID":"e3b1770a-983e-4a1f-8476-c37d1f5fe492","Type":"ContainerDied","Data":"1b4c29c66a51442e3f5ae37aa69962e64f1ada34f714b5f4ae940bd19f92a2ea"}
Nov 03 18:08:39.968946 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:39.968847    5315 scope.go:122] "RemoveContainer" containerID="960f0a4c0b29d0fc75234db263642b653380ab56a05a7739e30e42e48c99a083"
Nov 03 18:08:39.980643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:39.980622    5315 scope.go:122] "RemoveContainer" containerID="960f0a4c0b29d0fc75234db263642b653380ab56a05a7739e30e42e48c99a083"
Nov 03 18:08:39.980899 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:08:39.980879    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"960f0a4c0b29d0fc75234db263642b653380ab56a05a7739e30e42e48c99a083\": container with ID starting with 960f0a4c0b29d0fc75234db263642b653380ab56a05a7739e30e42e48c99a083 not found: ID does not exist" containerID="960f0a4c0b29d0fc75234db263642b653380ab56a05a7739e30e42e48c99a083"
Nov 03 18:08:39.980955 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:39.980906    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"960f0a4c0b29d0fc75234db263642b653380ab56a05a7739e30e42e48c99a083"} err="failed to get container status \"960f0a4c0b29d0fc75234db263642b653380ab56a05a7739e30e42e48c99a083\": rpc error: code = NotFound desc = could not find container \"960f0a4c0b29d0fc75234db263642b653380ab56a05a7739e30e42e48c99a083\": container with ID starting with 960f0a4c0b29d0fc75234db263642b653380ab56a05a7739e30e42e48c99a083 not found: ID does not exist"
Nov 03 18:08:40.192323 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:40.192285    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-86101f5d-78fb6dfcd7-4hcbt"]
Nov 03 18:08:40.202446 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:40.202408    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-86101f5d-78fb6dfcd7-4hcbt"]
Nov 03 18:08:40.256270 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:40.256187    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6"]
Nov 03 18:08:40.257307 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:40.257280    5315 cpu_manager.go:414] "containerMap: removing container" podUID="e3b1770a-983e-4a1f-8476-c37d1f5fe492" containerName="payload-pre-processing"
Nov 03 18:08:40.257414 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:40.257309    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3b1770a-983e-4a1f-8476-c37d1f5fe492" containerName="payload-pre-processing"
Nov 03 18:08:40.257545 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:40.257531    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3b1770a-983e-4a1f-8476-c37d1f5fe492" containerName="payload-pre-processing"
Nov 03 18:08:40.261072 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:40.261053    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6"
Nov 03 18:08:40.323302 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:40.323265    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6"]
Nov 03 18:08:40.361579 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:40.361536    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6511822b-0252-4d76-88c0-b4e27a0d2e6e-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-q2lb6\" (UID: \"6511822b-0252-4d76-88c0-b4e27a0d2e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6"
Nov 03 18:08:40.361800 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:40.361695    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6511822b-0252-4d76-88c0-b4e27a0d2e6e-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-q2lb6\" (UID: \"6511822b-0252-4d76-88c0-b4e27a0d2e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6"
Nov 03 18:08:40.361800 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:40.361739    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6511822b-0252-4d76-88c0-b4e27a0d2e6e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-q2lb6\" (UID: \"6511822b-0252-4d76-88c0-b4e27a0d2e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6"
Nov 03 18:08:40.361800 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:40.361775    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6511822b-0252-4d76-88c0-b4e27a0d2e6e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-q2lb6\" (UID: \"6511822b-0252-4d76-88c0-b4e27a0d2e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6"
Nov 03 18:08:40.361942 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:40.361903    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6511822b-0252-4d76-88c0-b4e27a0d2e6e-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-q2lb6\" (UID: \"6511822b-0252-4d76-88c0-b4e27a0d2e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6"
Nov 03 18:08:40.362013 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:40.361988    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6511822b-0252-4d76-88c0-b4e27a0d2e6e-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-q2lb6\" (UID: \"6511822b-0252-4d76-88c0-b4e27a0d2e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6"
Nov 03 18:08:40.362066 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:40.362046    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6llt8\" (UniqueName: \"kubernetes.io/projected/6511822b-0252-4d76-88c0-b4e27a0d2e6e-kube-api-access-6llt8\") pod \"maas-default-gateway-openshift-default-6dcf757899-q2lb6\" (UID: \"6511822b-0252-4d76-88c0-b4e27a0d2e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6"
Nov 03 18:08:40.362144 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:40.362127    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6511822b-0252-4d76-88c0-b4e27a0d2e6e-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-q2lb6\" (UID: \"6511822b-0252-4d76-88c0-b4e27a0d2e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6"
Nov 03 18:08:40.362250 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:40.362234    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6511822b-0252-4d76-88c0-b4e27a0d2e6e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-q2lb6\" (UID: \"6511822b-0252-4d76-88c0-b4e27a0d2e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6"
Nov 03 18:08:40.463663 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:40.463575    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6511822b-0252-4d76-88c0-b4e27a0d2e6e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-q2lb6\" (UID: \"6511822b-0252-4d76-88c0-b4e27a0d2e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6"
Nov 03 18:08:40.463841 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:40.463680    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6511822b-0252-4d76-88c0-b4e27a0d2e6e-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-q2lb6\" (UID: \"6511822b-0252-4d76-88c0-b4e27a0d2e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6"
Nov 03 18:08:40.463841 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:40.463729    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6511822b-0252-4d76-88c0-b4e27a0d2e6e-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-q2lb6\" (UID: \"6511822b-0252-4d76-88c0-b4e27a0d2e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6"
Nov 03 18:08:40.463841 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:40.463746    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6511822b-0252-4d76-88c0-b4e27a0d2e6e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-q2lb6\" (UID: \"6511822b-0252-4d76-88c0-b4e27a0d2e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6"
Nov 03 18:08:40.463987 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:40.463863    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6511822b-0252-4d76-88c0-b4e27a0d2e6e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-q2lb6\" (UID: \"6511822b-0252-4d76-88c0-b4e27a0d2e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6"
Nov 03 18:08:40.464048 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:40.464011    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6511822b-0252-4d76-88c0-b4e27a0d2e6e-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-q2lb6\" (UID: \"6511822b-0252-4d76-88c0-b4e27a0d2e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6"
Nov 03 18:08:40.464148 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:40.464105    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6511822b-0252-4d76-88c0-b4e27a0d2e6e-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-q2lb6\" (UID: \"6511822b-0252-4d76-88c0-b4e27a0d2e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6"
Nov 03 18:08:40.464206 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:40.464155    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6511822b-0252-4d76-88c0-b4e27a0d2e6e-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-q2lb6\" (UID: \"6511822b-0252-4d76-88c0-b4e27a0d2e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6"
Nov 03 18:08:40.464206 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:40.464162    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6llt8\" (UniqueName: \"kubernetes.io/projected/6511822b-0252-4d76-88c0-b4e27a0d2e6e-kube-api-access-6llt8\") pod \"maas-default-gateway-openshift-default-6dcf757899-q2lb6\" (UID: \"6511822b-0252-4d76-88c0-b4e27a0d2e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6"
Nov 03 18:08:40.464331 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:40.464218    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6511822b-0252-4d76-88c0-b4e27a0d2e6e-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-q2lb6\" (UID: \"6511822b-0252-4d76-88c0-b4e27a0d2e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6"
Nov 03 18:08:40.464331 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:40.464253    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6511822b-0252-4d76-88c0-b4e27a0d2e6e-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-q2lb6\" (UID: \"6511822b-0252-4d76-88c0-b4e27a0d2e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6"
Nov 03 18:08:40.464331 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:40.464309    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6511822b-0252-4d76-88c0-b4e27a0d2e6e-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-q2lb6\" (UID: \"6511822b-0252-4d76-88c0-b4e27a0d2e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6"
Nov 03 18:08:40.464487 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:40.464406    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6511822b-0252-4d76-88c0-b4e27a0d2e6e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-q2lb6\" (UID: \"6511822b-0252-4d76-88c0-b4e27a0d2e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6"
Nov 03 18:08:40.464539 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:40.464517    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6511822b-0252-4d76-88c0-b4e27a0d2e6e-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-q2lb6\" (UID: \"6511822b-0252-4d76-88c0-b4e27a0d2e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6"
Nov 03 18:08:40.466310 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:40.466282    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6511822b-0252-4d76-88c0-b4e27a0d2e6e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-q2lb6\" (UID: \"6511822b-0252-4d76-88c0-b4e27a0d2e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6"
Nov 03 18:08:40.466461 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:40.466441    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6511822b-0252-4d76-88c0-b4e27a0d2e6e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-q2lb6\" (UID: \"6511822b-0252-4d76-88c0-b4e27a0d2e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6"
Nov 03 18:08:40.521671 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:40.521583    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6511822b-0252-4d76-88c0-b4e27a0d2e6e-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-q2lb6\" (UID: \"6511822b-0252-4d76-88c0-b4e27a0d2e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6"
Nov 03 18:08:40.521799 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:40.521755    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6llt8\" (UniqueName: \"kubernetes.io/projected/6511822b-0252-4d76-88c0-b4e27a0d2e6e-kube-api-access-6llt8\") pod \"maas-default-gateway-openshift-default-6dcf757899-q2lb6\" (UID: \"6511822b-0252-4d76-88c0-b4e27a0d2e6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6"
Nov 03 18:08:40.574175 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:40.574136    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6"
Nov 03 18:08:40.781137 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:40.781096    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q"
Nov 03 18:08:40.781302 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:40.781283    5315 patch_prober.go:29] interesting pod/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 03 18:08:40.781352 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:40.781334    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q" podUID="11671eb9-4824-4e89-9692-b5be9038c549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 03 18:08:41.055389 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:41.055358    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6"]
Nov 03 18:08:41.056451 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:08:41.056426    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6511822b_0252_4d76_88c0_b4e27a0d2e6e.slice/crio-6c1e310cc0667809c386471c2a422cbe98bd85e571da92b40c0731343857fc2b WatchSource:0}: Error finding container 6c1e310cc0667809c386471c2a422cbe98bd85e571da92b40c0731343857fc2b: Status 404 returned error can't find the container with id 6c1e310cc0667809c386471c2a422cbe98bd85e571da92b40c0731343857fc2b
Nov 03 18:08:41.059712 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:41.059598    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:08:41.059712 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:41.059706    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:08:41.059820 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:41.059748    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:08:41.293368 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:41.293333    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3b1770a-983e-4a1f-8476-c37d1f5fe492" path="/var/lib/kubelet/pods/e3b1770a-983e-4a1f-8476-c37d1f5fe492/volumes"
Nov 03 18:08:41.368516 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:41.368487    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-29bb4e29-42cgh"
Nov 03 18:08:41.407808 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:41.407777    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d05f048d-1346-491b-bc11-121a6c7d53a6-kube-api-access-zpg8j\" (UniqueName: \"kubernetes.io/projected/d05f048d-1346-491b-bc11-121a6c7d53a6-kube-api-access-zpg8j\") pod \"d05f048d-1346-491b-bc11-121a6c7d53a6\" (UID: \"d05f048d-1346-491b-bc11-121a6c7d53a6\") "
Nov 03 18:08:41.407994 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:41.407858    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d05f048d-1346-491b-bc11-121a6c7d53a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d05f048d-1346-491b-bc11-121a6c7d53a6-maas-api-xks-ca\") pod \"d05f048d-1346-491b-bc11-121a6c7d53a6\" (UID: \"d05f048d-1346-491b-bc11-121a6c7d53a6\") "
Nov 03 18:08:41.408184 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:41.408164    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d05f048d-1346-491b-bc11-121a6c7d53a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d05f048d-1346-491b-bc11-121a6c7d53a6-maas-api-openshift-ca\") pod \"d05f048d-1346-491b-bc11-121a6c7d53a6\" (UID: \"d05f048d-1346-491b-bc11-121a6c7d53a6\") "
Nov 03 18:08:41.408562 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:41.408533    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d05f048d-1346-491b-bc11-121a6c7d53a6-maas-api-openshift-ca" pod "d05f048d-1346-491b-bc11-121a6c7d53a6" (UID: "d05f048d-1346-491b-bc11-121a6c7d53a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:08:41.409424 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:41.409399    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d05f048d-1346-491b-bc11-121a6c7d53a6-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:08:41.410301 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:41.410113    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d05f048d-1346-491b-bc11-121a6c7d53a6-kube-api-access-zpg8j" pod "d05f048d-1346-491b-bc11-121a6c7d53a6" (UID: "d05f048d-1346-491b-bc11-121a6c7d53a6"). InnerVolumeSpecName "kube-api-access-zpg8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:08:41.410510 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:41.410489    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d05f048d-1346-491b-bc11-121a6c7d53a6-maas-api-xks-ca" pod "d05f048d-1346-491b-bc11-121a6c7d53a6" (UID: "d05f048d-1346-491b-bc11-121a6c7d53a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:08:41.511041 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:41.511004    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d05f048d-1346-491b-bc11-121a6c7d53a6-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:08:41.511191 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:41.511045    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpg8j\" (UniqueName: \"kubernetes.io/projected/d05f048d-1346-491b-bc11-121a6c7d53a6-kube-api-access-zpg8j\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:08:41.781229 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:41.781196    5315 patch_prober.go:29] interesting pod/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 03 18:08:41.781396 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:41.781263    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q" podUID="11671eb9-4824-4e89-9692-b5be9038c549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 03 18:08:41.989469 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:41.989384    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-29bb4e29-42cgh"
Nov 03 18:08:41.989469 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:41.989422    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-29bb4e29-42cgh" event={"ID":"d05f048d-1346-491b-bc11-121a6c7d53a6","Type":"ContainerDied","Data":"99118c3dfc18aab2a5ee4a4354ef7fc858feddc9b8732e24854aebfbd79d1de8"}
Nov 03 18:08:41.989759 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:41.989485    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99118c3dfc18aab2a5ee4a4354ef7fc858feddc9b8732e24854aebfbd79d1de8"
Nov 03 18:08:41.991886 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:41.991855    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6" event={"ID":"6511822b-0252-4d76-88c0-b4e27a0d2e6e","Type":"ContainerStarted","Data":"7620550b4f502e4495db5d545e3614b0bf45ffe303b473e8d496eed5c5fe6eae"}
Nov 03 18:08:41.992023 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:41.991921    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6" event={"ID":"6511822b-0252-4d76-88c0-b4e27a0d2e6e","Type":"ContainerStarted","Data":"6c1e310cc0667809c386471c2a422cbe98bd85e571da92b40c0731343857fc2b"}
Nov 03 18:08:42.118168 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:42.118011    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6" podStartSLOduration=2.117988442 podStartE2EDuration="2.117988442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:08:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:08:42.114007411 +0000 UTC m=+1361.130205461" watchObservedRunningTime="2026-11-03 18:08:42.117988442 +0000 UTC m=+1361.134186510"
Nov 03 18:08:42.577990 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:42.577723    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6"
Nov 03 18:08:42.578725 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:42.578699    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-q2lb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 03 18:08:42.578925 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:42.578900    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6" podUID="6511822b-0252-4d76-88c0-b4e27a0d2e6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 03 18:08:42.781717 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:42.781678    5315 patch_prober.go:29] interesting pod/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 03 18:08:42.781902 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:42.781750    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q" podUID="11671eb9-4824-4e89-9692-b5be9038c549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 03 18:08:43.574717 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:43.574546    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-q2lb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 03 18:08:43.574717 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:43.574650    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6" podUID="6511822b-0252-4d76-88c0-b4e27a0d2e6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 03 18:08:43.781086 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:43.781046    5315 patch_prober.go:29] interesting pod/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 03 18:08:43.781263 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:43.781099    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q" podUID="11671eb9-4824-4e89-9692-b5be9038c549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 03 18:08:44.576020 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:44.575978    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-q2lb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 03 18:08:44.577077 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:44.576650    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6" podUID="6511822b-0252-4d76-88c0-b4e27a0d2e6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 03 18:08:44.781978 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:44.781945    5315 patch_prober.go:29] interesting pod/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 03 18:08:44.782179 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:44.782039    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q" podUID="11671eb9-4824-4e89-9692-b5be9038c549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 03 18:08:45.575625 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:45.574986    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-q2lb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 03 18:08:45.575625 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:45.575049    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6" podUID="6511822b-0252-4d76-88c0-b4e27a0d2e6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 03 18:08:45.785857 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:45.785827    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q"
Nov 03 18:08:45.786367 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:45.786345    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q"
Nov 03 18:08:45.787081 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:45.787058    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q"
Nov 03 18:08:46.574879 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:46.574835    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-q2lb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 03 18:08:46.575102 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:46.574896    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6" podUID="6511822b-0252-4d76-88c0-b4e27a0d2e6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 03 18:08:47.575139 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:47.575099    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-q2lb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 03 18:08:47.575517 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:47.575168    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6" podUID="6511822b-0252-4d76-88c0-b4e27a0d2e6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 03 18:08:48.575308 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:48.575265    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-q2lb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 03 18:08:48.575797 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:48.575339    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6" podUID="6511822b-0252-4d76-88c0-b4e27a0d2e6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 03 18:08:50.176636 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:50.173970    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6"
Nov 03 18:08:50.176636 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:50.175693    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6"
Nov 03 18:08:50.185905 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:50.185868    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6"
Nov 03 18:08:54.810242 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:08:54.810194    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:08:55.393924 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:55.393879    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89"]
Nov 03 18:08:55.395425 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:55.395312    5315 cpu_manager.go:414] "containerMap: removing container" podUID="d05f048d-1346-491b-bc11-121a6c7d53a6" containerName="revoke-keys"
Nov 03 18:08:55.395716 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:55.395585    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d05f048d-1346-491b-bc11-121a6c7d53a6" containerName="revoke-keys"
Nov 03 18:08:55.395852 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:55.395844    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="d05f048d-1346-491b-bc11-121a6c7d53a6" containerName="revoke-keys"
Nov 03 18:08:55.399816 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:55.399791    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89"
Nov 03 18:08:55.449976 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:55.449937    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16411400-4aa7-4754-9dc2-82175924011a-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-6gf89\" (UID: \"16411400-4aa7-4754-9dc2-82175924011a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89"
Nov 03 18:08:55.450183 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:55.450108    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16411400-4aa7-4754-9dc2-82175924011a-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-6gf89\" (UID: \"16411400-4aa7-4754-9dc2-82175924011a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89"
Nov 03 18:08:55.450183 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:55.450140    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16411400-4aa7-4754-9dc2-82175924011a-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-6gf89\" (UID: \"16411400-4aa7-4754-9dc2-82175924011a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89"
Nov 03 18:08:55.450320 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:55.450280    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16411400-4aa7-4754-9dc2-82175924011a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-6gf89\" (UID: \"16411400-4aa7-4754-9dc2-82175924011a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89"
Nov 03 18:08:55.450428 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:55.450406    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16411400-4aa7-4754-9dc2-82175924011a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-6gf89\" (UID: \"16411400-4aa7-4754-9dc2-82175924011a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89"
Nov 03 18:08:55.450468 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:55.450457    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16411400-4aa7-4754-9dc2-82175924011a-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-6gf89\" (UID: \"16411400-4aa7-4754-9dc2-82175924011a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89"
Nov 03 18:08:55.450649 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:55.450589    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16411400-4aa7-4754-9dc2-82175924011a-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-6gf89\" (UID: \"16411400-4aa7-4754-9dc2-82175924011a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89"
Nov 03 18:08:55.450769 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:55.450752    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16411400-4aa7-4754-9dc2-82175924011a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-6gf89\" (UID: \"16411400-4aa7-4754-9dc2-82175924011a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89"
Nov 03 18:08:55.450840 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:55.450823    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jrz6\" (UniqueName: \"kubernetes.io/projected/16411400-4aa7-4754-9dc2-82175924011a-kube-api-access-8jrz6\") pod \"maas-default-gateway-openshift-default-6dcf757899-6gf89\" (UID: \"16411400-4aa7-4754-9dc2-82175924011a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89"
Nov 03 18:08:55.480599 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:55.480547    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89"]
Nov 03 18:08:55.552737 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:55.552676    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16411400-4aa7-4754-9dc2-82175924011a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-6gf89\" (UID: \"16411400-4aa7-4754-9dc2-82175924011a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89"
Nov 03 18:08:55.552942 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:55.552770    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jrz6\" (UniqueName: \"kubernetes.io/projected/16411400-4aa7-4754-9dc2-82175924011a-kube-api-access-8jrz6\") pod \"maas-default-gateway-openshift-default-6dcf757899-6gf89\" (UID: \"16411400-4aa7-4754-9dc2-82175924011a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89"
Nov 03 18:08:55.552942 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:55.552852    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16411400-4aa7-4754-9dc2-82175924011a-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-6gf89\" (UID: \"16411400-4aa7-4754-9dc2-82175924011a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89"
Nov 03 18:08:55.553080 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:55.552975    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16411400-4aa7-4754-9dc2-82175924011a-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-6gf89\" (UID: \"16411400-4aa7-4754-9dc2-82175924011a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89"
Nov 03 18:08:55.553080 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:55.553005    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16411400-4aa7-4754-9dc2-82175924011a-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-6gf89\" (UID: \"16411400-4aa7-4754-9dc2-82175924011a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89"
Nov 03 18:08:55.553080 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:55.553062    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16411400-4aa7-4754-9dc2-82175924011a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-6gf89\" (UID: \"16411400-4aa7-4754-9dc2-82175924011a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89"
Nov 03 18:08:55.553268 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:55.553150    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16411400-4aa7-4754-9dc2-82175924011a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-6gf89\" (UID: \"16411400-4aa7-4754-9dc2-82175924011a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89"
Nov 03 18:08:55.553268 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:55.553194    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16411400-4aa7-4754-9dc2-82175924011a-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-6gf89\" (UID: \"16411400-4aa7-4754-9dc2-82175924011a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89"
Nov 03 18:08:55.553380 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:55.553279    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16411400-4aa7-4754-9dc2-82175924011a-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-6gf89\" (UID: \"16411400-4aa7-4754-9dc2-82175924011a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89"
Nov 03 18:08:55.553380 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:55.553287    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16411400-4aa7-4754-9dc2-82175924011a-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-6gf89\" (UID: \"16411400-4aa7-4754-9dc2-82175924011a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89"
Nov 03 18:08:55.553495 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:55.553422    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16411400-4aa7-4754-9dc2-82175924011a-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-6gf89\" (UID: \"16411400-4aa7-4754-9dc2-82175924011a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89"
Nov 03 18:08:55.553693 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:55.553667    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16411400-4aa7-4754-9dc2-82175924011a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-6gf89\" (UID: \"16411400-4aa7-4754-9dc2-82175924011a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89"
Nov 03 18:08:55.553789 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:55.553668    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16411400-4aa7-4754-9dc2-82175924011a-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-6gf89\" (UID: \"16411400-4aa7-4754-9dc2-82175924011a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89"
Nov 03 18:08:55.553861 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:55.553815    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16411400-4aa7-4754-9dc2-82175924011a-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-6gf89\" (UID: \"16411400-4aa7-4754-9dc2-82175924011a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89"
Nov 03 18:08:55.555560 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:55.555528    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16411400-4aa7-4754-9dc2-82175924011a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-6gf89\" (UID: \"16411400-4aa7-4754-9dc2-82175924011a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89"
Nov 03 18:08:55.555696 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:55.555590    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16411400-4aa7-4754-9dc2-82175924011a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-6gf89\" (UID: \"16411400-4aa7-4754-9dc2-82175924011a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89"
Nov 03 18:08:55.609887 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:55.609850    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16411400-4aa7-4754-9dc2-82175924011a-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-6gf89\" (UID: \"16411400-4aa7-4754-9dc2-82175924011a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89"
Nov 03 18:08:55.609887 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:55.609866    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jrz6\" (UniqueName: \"kubernetes.io/projected/16411400-4aa7-4754-9dc2-82175924011a-kube-api-access-8jrz6\") pod \"maas-default-gateway-openshift-default-6dcf757899-6gf89\" (UID: \"16411400-4aa7-4754-9dc2-82175924011a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89"
Nov 03 18:08:55.716859 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:55.716770    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89"
Nov 03 18:08:56.150726 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:56.150673    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89"]
Nov 03 18:08:56.153578 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:08:56.153545    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16411400_4aa7_4754_9dc2_82175924011a.slice/crio-7b0cfa490a5bc46dbd6ad4fa8d7331d04d2be788758e382c74f55ad85440c91d WatchSource:0}: Error finding container 7b0cfa490a5bc46dbd6ad4fa8d7331d04d2be788758e382c74f55ad85440c91d: Status 404 returned error can't find the container with id 7b0cfa490a5bc46dbd6ad4fa8d7331d04d2be788758e382c74f55ad85440c91d
Nov 03 18:08:56.156455 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:56.156422    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:08:56.156551 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:56.156523    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:08:56.156600 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:56.156565    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:08:56.410346 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:56.410254    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89" event={"ID":"16411400-4aa7-4754-9dc2-82175924011a","Type":"ContainerStarted","Data":"5c9d793dda22c3f9f174984911ac0248444bfb8cf3ac61f4e371214658b2a8d8"}
Nov 03 18:08:56.410492 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:56.410369    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89" event={"ID":"16411400-4aa7-4754-9dc2-82175924011a","Type":"ContainerStarted","Data":"7b0cfa490a5bc46dbd6ad4fa8d7331d04d2be788758e382c74f55ad85440c91d"}
Nov 03 18:08:56.529689 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:56.529598    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89" podStartSLOduration=1.529579679 podStartE2EDuration="1.529579679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:08:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:08:56.522058977 +0000 UTC m=+1375.538257046" watchObservedRunningTime="2026-11-03 18:08:56.529579679 +0000 UTC m=+1375.545777750"
Nov 03 18:08:56.716971 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:56.716890    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89"
Nov 03 18:08:57.719484 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:57.719444    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-6gf89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 03 18:08:57.719879 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:57.719520    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89" podUID="16411400-4aa7-4754-9dc2-82175924011a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 03 18:08:58.717469 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:58.717431    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-6gf89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 03 18:08:58.717667 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:58.717492    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89" podUID="16411400-4aa7-4754-9dc2-82175924011a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 03 18:08:59.717599 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:59.717560    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-6gf89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 03 18:08:59.718074 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:08:59.717664    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89" podUID="16411400-4aa7-4754-9dc2-82175924011a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 03 18:09:00.717560 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:00.717522    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-6gf89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 03 18:09:00.717760 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:00.717577    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89" podUID="16411400-4aa7-4754-9dc2-82175924011a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 03 18:09:01.151491 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:01.151437    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-248f8ee3-rvrks"]
Nov 03 18:09:01.160497 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:01.160464    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-248f8ee3-rvrks"
Nov 03 18:09:01.196267 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:01.196236    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-248f8ee3-rvrks"]
Nov 03 18:09:01.250183 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:01.250133    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:09:01.250393 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:01.250212    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:09:01.261195 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:01.261168    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:09:01.261357 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:01.261197    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:09:01.278122 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:01.278082    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/522d42d2-228a-4242-bd5f-b2e3074b7b89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-248f8ee3-rvrks\" (UID: \"522d42d2-228a-4242-bd5f-b2e3074b7b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-248f8ee3-rvrks"
Nov 03 18:09:01.278122 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:01.278134    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/522d42d2-228a-4242-bd5f-b2e3074b7b89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-248f8ee3-rvrks\" (UID: \"522d42d2-228a-4242-bd5f-b2e3074b7b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-248f8ee3-rvrks"
Nov 03 18:09:01.278605 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:01.278314    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrtwp\" (UniqueName: \"kubernetes.io/projected/522d42d2-228a-4242-bd5f-b2e3074b7b89-kube-api-access-xrtwp\") pod \"maas-api-revoke-keys-e2e-ait-248f8ee3-rvrks\" (UID: \"522d42d2-228a-4242-bd5f-b2e3074b7b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-248f8ee3-rvrks"
Nov 03 18:09:01.379721 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:01.379685    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/522d42d2-228a-4242-bd5f-b2e3074b7b89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-248f8ee3-rvrks\" (UID: \"522d42d2-228a-4242-bd5f-b2e3074b7b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-248f8ee3-rvrks"
Nov 03 18:09:01.379721 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:01.379724    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/522d42d2-228a-4242-bd5f-b2e3074b7b89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-248f8ee3-rvrks\" (UID: \"522d42d2-228a-4242-bd5f-b2e3074b7b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-248f8ee3-rvrks"
Nov 03 18:09:01.379965 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:01.379795    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrtwp\" (UniqueName: \"kubernetes.io/projected/522d42d2-228a-4242-bd5f-b2e3074b7b89-kube-api-access-xrtwp\") pod \"maas-api-revoke-keys-e2e-ait-248f8ee3-rvrks\" (UID: \"522d42d2-228a-4242-bd5f-b2e3074b7b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-248f8ee3-rvrks"
Nov 03 18:09:01.380480 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:01.380455    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/522d42d2-228a-4242-bd5f-b2e3074b7b89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-248f8ee3-rvrks\" (UID: \"522d42d2-228a-4242-bd5f-b2e3074b7b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-248f8ee3-rvrks"
Nov 03 18:09:01.382075 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:01.382048    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/522d42d2-228a-4242-bd5f-b2e3074b7b89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-248f8ee3-rvrks\" (UID: \"522d42d2-228a-4242-bd5f-b2e3074b7b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-248f8ee3-rvrks"
Nov 03 18:09:01.455484 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:01.455411    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrtwp\" (UniqueName: \"kubernetes.io/projected/522d42d2-228a-4242-bd5f-b2e3074b7b89-kube-api-access-xrtwp\") pod \"maas-api-revoke-keys-e2e-ait-248f8ee3-rvrks\" (UID: \"522d42d2-228a-4242-bd5f-b2e3074b7b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-248f8ee3-rvrks"
Nov 03 18:09:01.475165 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:01.475138    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-248f8ee3-rvrks"
Nov 03 18:09:01.718093 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:01.718005    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-6gf89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 03 18:09:01.718093 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:01.718064    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89" podUID="16411400-4aa7-4754-9dc2-82175924011a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 03 18:09:01.879131 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:01.879100    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-248f8ee3-rvrks"]
Nov 03 18:09:01.881820 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:09:01.881785    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod522d42d2_228a_4242_bd5f_b2e3074b7b89.slice/crio-8687bd5b5a95c6b021a664073fd66ea8b9216a3933fbd26109c10d3b8769c334 WatchSource:0}: Error finding container 8687bd5b5a95c6b021a664073fd66ea8b9216a3933fbd26109c10d3b8769c334: Status 404 returned error can't find the container with id 8687bd5b5a95c6b021a664073fd66ea8b9216a3933fbd26109c10d3b8769c334
Nov 03 18:09:02.493540 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:02.493501    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-248f8ee3-rvrks" event={"ID":"522d42d2-228a-4242-bd5f-b2e3074b7b89","Type":"ContainerStarted","Data":"8bb6a8719a14e5b36a0e8736898d047f4fcb42e2a4258f42617d7020447b895f"}
Nov 03 18:09:02.493748 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:02.493573    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-248f8ee3-rvrks" event={"ID":"522d42d2-228a-4242-bd5f-b2e3074b7b89","Type":"ContainerStarted","Data":"8687bd5b5a95c6b021a664073fd66ea8b9216a3933fbd26109c10d3b8769c334"}
Nov 03 18:09:02.609602 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:02.609496    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-248f8ee3-rvrks" podStartSLOduration=1.609481822 podStartE2EDuration="1.609481822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:09:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:09:02.607152406 +0000 UTC m=+1381.623350474" watchObservedRunningTime="2026-11-03 18:09:02.609481822 +0000 UTC m=+1381.625679892"
Nov 03 18:09:02.717859 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:02.717815    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-6gf89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 03 18:09:02.718042 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:02.717897    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89" podUID="16411400-4aa7-4754-9dc2-82175924011a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 03 18:09:03.503168 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:03.503131    5315 generic.go:356] "Generic (PLEG): container finished" podID="522d42d2-228a-4242-bd5f-b2e3074b7b89" containerID="8bb6a8719a14e5b36a0e8736898d047f4fcb42e2a4258f42617d7020447b895f" exitCode=0
Nov 03 18:09:03.503540 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:03.503218    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-248f8ee3-rvrks" event={"ID":"522d42d2-228a-4242-bd5f-b2e3074b7b89","Type":"ContainerDied","Data":"8bb6a8719a14e5b36a0e8736898d047f4fcb42e2a4258f42617d7020447b895f"}
Nov 03 18:09:03.718092 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:03.718056    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-6gf89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 03 18:09:03.718260 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:03.718115    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89" podUID="16411400-4aa7-4754-9dc2-82175924011a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 03 18:09:05.082477 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:05.082437    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-248f8ee3-rvrks"
Nov 03 18:09:05.088184 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:05.088154    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/522d42d2-228a-4242-bd5f-b2e3074b7b89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/522d42d2-228a-4242-bd5f-b2e3074b7b89-maas-api-openshift-ca\") pod \"522d42d2-228a-4242-bd5f-b2e3074b7b89\" (UID: \"522d42d2-228a-4242-bd5f-b2e3074b7b89\") "
Nov 03 18:09:05.088335 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:05.088315    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/522d42d2-228a-4242-bd5f-b2e3074b7b89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/522d42d2-228a-4242-bd5f-b2e3074b7b89-maas-api-xks-ca\") pod \"522d42d2-228a-4242-bd5f-b2e3074b7b89\" (UID: \"522d42d2-228a-4242-bd5f-b2e3074b7b89\") "
Nov 03 18:09:05.088473 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:05.088456    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/522d42d2-228a-4242-bd5f-b2e3074b7b89-kube-api-access-xrtwp\" (UniqueName: \"kubernetes.io/projected/522d42d2-228a-4242-bd5f-b2e3074b7b89-kube-api-access-xrtwp\") pod \"522d42d2-228a-4242-bd5f-b2e3074b7b89\" (UID: \"522d42d2-228a-4242-bd5f-b2e3074b7b89\") "
Nov 03 18:09:05.088594 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:05.088564    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/522d42d2-228a-4242-bd5f-b2e3074b7b89-maas-api-openshift-ca" pod "522d42d2-228a-4242-bd5f-b2e3074b7b89" (UID: "522d42d2-228a-4242-bd5f-b2e3074b7b89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:09:05.089383 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:05.089357    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/522d42d2-228a-4242-bd5f-b2e3074b7b89-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:05.090604 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:05.090573    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/522d42d2-228a-4242-bd5f-b2e3074b7b89-kube-api-access-xrtwp" pod "522d42d2-228a-4242-bd5f-b2e3074b7b89" (UID: "522d42d2-228a-4242-bd5f-b2e3074b7b89"). InnerVolumeSpecName "kube-api-access-xrtwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:09:05.091024 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:05.091000    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/522d42d2-228a-4242-bd5f-b2e3074b7b89-maas-api-xks-ca" pod "522d42d2-228a-4242-bd5f-b2e3074b7b89" (UID: "522d42d2-228a-4242-bd5f-b2e3074b7b89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:09:05.195701 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:05.191049    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/522d42d2-228a-4242-bd5f-b2e3074b7b89-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:05.195701 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:05.191084    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrtwp\" (UniqueName: \"kubernetes.io/projected/522d42d2-228a-4242-bd5f-b2e3074b7b89-kube-api-access-xrtwp\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:05.565647 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:05.565584    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-248f8ee3-rvrks"
Nov 03 18:09:05.565825 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:05.565586    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-248f8ee3-rvrks" event={"ID":"522d42d2-228a-4242-bd5f-b2e3074b7b89","Type":"ContainerDied","Data":"8687bd5b5a95c6b021a664073fd66ea8b9216a3933fbd26109c10d3b8769c334"}
Nov 03 18:09:05.565825 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:05.565771    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8687bd5b5a95c6b021a664073fd66ea8b9216a3933fbd26109c10d3b8769c334"
Nov 03 18:09:05.582118 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:05.582086    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89"
Nov 03 18:09:05.582709 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:05.582670    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89"
Nov 03 18:09:05.583685 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:05.583666    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89"
Nov 03 18:09:07.897695 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:07.895837    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b6e7259-fwzg9"]
Nov 03 18:09:07.897695 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:07.897108    5315 cpu_manager.go:414] "containerMap: removing container" podUID="522d42d2-228a-4242-bd5f-b2e3074b7b89" containerName="revoke-keys"
Nov 03 18:09:07.897695 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:07.897128    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="522d42d2-228a-4242-bd5f-b2e3074b7b89" containerName="revoke-keys"
Nov 03 18:09:07.897695 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:07.897260    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="522d42d2-228a-4242-bd5f-b2e3074b7b89" containerName="revoke-keys"
Nov 03 18:09:07.904213 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:07.904182    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b6e7259-fwzg9"
Nov 03 18:09:07.925215 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:07.925184    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b6e7259-fwzg9"]
Nov 03 18:09:07.986630 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:07.986578    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/885b7b64-07d4-4d24-b961-77841e1f4ba8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4b6e7259-fwzg9\" (UID: \"885b7b64-07d4-4d24-b961-77841e1f4ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b6e7259-fwzg9"
Nov 03 18:09:07.986869 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:07.986842    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9kxk\" (UniqueName: \"kubernetes.io/projected/885b7b64-07d4-4d24-b961-77841e1f4ba8-kube-api-access-c9kxk\") pod \"maas-api-revoke-keys-e2e-ait-4b6e7259-fwzg9\" (UID: \"885b7b64-07d4-4d24-b961-77841e1f4ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b6e7259-fwzg9"
Nov 03 18:09:07.987350 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:07.987325    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/885b7b64-07d4-4d24-b961-77841e1f4ba8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4b6e7259-fwzg9\" (UID: \"885b7b64-07d4-4d24-b961-77841e1f4ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b6e7259-fwzg9"
Nov 03 18:09:07.996081 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:07.996040    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:09:07.996373 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:07.996354    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:09:07.996655 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:07.996637    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:09:08.032058 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:08.032024    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:09:08.089435 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:08.089346    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/885b7b64-07d4-4d24-b961-77841e1f4ba8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4b6e7259-fwzg9\" (UID: \"885b7b64-07d4-4d24-b961-77841e1f4ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b6e7259-fwzg9"
Nov 03 18:09:08.090348 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:08.090321    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/885b7b64-07d4-4d24-b961-77841e1f4ba8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4b6e7259-fwzg9\" (UID: \"885b7b64-07d4-4d24-b961-77841e1f4ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b6e7259-fwzg9"
Nov 03 18:09:08.090545 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:08.090525    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9kxk\" (UniqueName: \"kubernetes.io/projected/885b7b64-07d4-4d24-b961-77841e1f4ba8-kube-api-access-c9kxk\") pod \"maas-api-revoke-keys-e2e-ait-4b6e7259-fwzg9\" (UID: \"885b7b64-07d4-4d24-b961-77841e1f4ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b6e7259-fwzg9"
Nov 03 18:09:08.092657 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:08.091694    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/885b7b64-07d4-4d24-b961-77841e1f4ba8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4b6e7259-fwzg9\" (UID: \"885b7b64-07d4-4d24-b961-77841e1f4ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b6e7259-fwzg9"
Nov 03 18:09:08.093250 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:08.093227    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/885b7b64-07d4-4d24-b961-77841e1f4ba8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4b6e7259-fwzg9\" (UID: \"885b7b64-07d4-4d24-b961-77841e1f4ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b6e7259-fwzg9"
Nov 03 18:09:08.152262 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:08.151962    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9kxk\" (UniqueName: \"kubernetes.io/projected/885b7b64-07d4-4d24-b961-77841e1f4ba8-kube-api-access-c9kxk\") pod \"maas-api-revoke-keys-e2e-ait-4b6e7259-fwzg9\" (UID: \"885b7b64-07d4-4d24-b961-77841e1f4ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b6e7259-fwzg9"
Nov 03 18:09:08.219570 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:08.219533    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b6e7259-fwzg9"
Nov 03 18:09:08.740719 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:08.740664    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b6e7259-fwzg9"]
Nov 03 18:09:09.634562 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:09.634528    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b6e7259-fwzg9" event={"ID":"885b7b64-07d4-4d24-b961-77841e1f4ba8","Type":"ContainerStarted","Data":"96be76605e544234a34e3708facc81660dd82760f6c083366e7e318c12a3c55a"}
Nov 03 18:09:09.635009 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:09.634585    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b6e7259-fwzg9" event={"ID":"885b7b64-07d4-4d24-b961-77841e1f4ba8","Type":"ContainerStarted","Data":"e44720ae1ae1d75575e6cc8881d44cac317ef51331347746668a5aef57383003"}
Nov 03 18:09:09.747841 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:09.747780    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b6e7259-fwzg9" podStartSLOduration=2.747760447 podStartE2EDuration="2.747760447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:09:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:09:09.743676199 +0000 UTC m=+1388.759874273" watchObservedRunningTime="2026-11-03 18:09:09.747760447 +0000 UTC m=+1388.763958516"
Nov 03 18:09:10.644053 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:10.644016    5315 generic.go:356] "Generic (PLEG): container finished" podID="885b7b64-07d4-4d24-b961-77841e1f4ba8" containerID="96be76605e544234a34e3708facc81660dd82760f6c083366e7e318c12a3c55a" exitCode=0
Nov 03 18:09:10.644527 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:10.644113    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b6e7259-fwzg9" event={"ID":"885b7b64-07d4-4d24-b961-77841e1f4ba8","Type":"ContainerDied","Data":"96be76605e544234a34e3708facc81660dd82760f6c083366e7e318c12a3c55a"}
Nov 03 18:09:11.854060 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:11.854033    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b6e7259-fwzg9"
Nov 03 18:09:11.976707 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:11.976569    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/885b7b64-07d4-4d24-b961-77841e1f4ba8-kube-api-access-c9kxk\" (UniqueName: \"kubernetes.io/projected/885b7b64-07d4-4d24-b961-77841e1f4ba8-kube-api-access-c9kxk\") pod \"885b7b64-07d4-4d24-b961-77841e1f4ba8\" (UID: \"885b7b64-07d4-4d24-b961-77841e1f4ba8\") "
Nov 03 18:09:11.976889 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:11.976755    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/885b7b64-07d4-4d24-b961-77841e1f4ba8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/885b7b64-07d4-4d24-b961-77841e1f4ba8-maas-api-openshift-ca\") pod \"885b7b64-07d4-4d24-b961-77841e1f4ba8\" (UID: \"885b7b64-07d4-4d24-b961-77841e1f4ba8\") "
Nov 03 18:09:11.976889 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:11.976857    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/885b7b64-07d4-4d24-b961-77841e1f4ba8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/885b7b64-07d4-4d24-b961-77841e1f4ba8-maas-api-xks-ca\") pod \"885b7b64-07d4-4d24-b961-77841e1f4ba8\" (UID: \"885b7b64-07d4-4d24-b961-77841e1f4ba8\") "
Nov 03 18:09:11.977118 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:11.977090    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/885b7b64-07d4-4d24-b961-77841e1f4ba8-maas-api-openshift-ca" pod "885b7b64-07d4-4d24-b961-77841e1f4ba8" (UID: "885b7b64-07d4-4d24-b961-77841e1f4ba8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:09:11.977560 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:11.977536    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/885b7b64-07d4-4d24-b961-77841e1f4ba8-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:11.979019 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:11.978975    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/885b7b64-07d4-4d24-b961-77841e1f4ba8-maas-api-xks-ca" pod "885b7b64-07d4-4d24-b961-77841e1f4ba8" (UID: "885b7b64-07d4-4d24-b961-77841e1f4ba8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:09:11.979019 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:11.978981    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/885b7b64-07d4-4d24-b961-77841e1f4ba8-kube-api-access-c9kxk" pod "885b7b64-07d4-4d24-b961-77841e1f4ba8" (UID: "885b7b64-07d4-4d24-b961-77841e1f4ba8"). InnerVolumeSpecName "kube-api-access-c9kxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:09:12.079059 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:12.079014    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9kxk\" (UniqueName: \"kubernetes.io/projected/885b7b64-07d4-4d24-b961-77841e1f4ba8-kube-api-access-c9kxk\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:12.079059 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:12.079053    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/885b7b64-07d4-4d24-b961-77841e1f4ba8-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:12.656423 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:12.656379    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-248f8ee3-574c4c5b5-rqjv5"]
Nov 03 18:09:12.667759 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:12.667730    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b6e7259-fwzg9"
Nov 03 18:09:12.667941 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:12.667768    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b6e7259-fwzg9" event={"ID":"885b7b64-07d4-4d24-b961-77841e1f4ba8","Type":"ContainerDied","Data":"e44720ae1ae1d75575e6cc8881d44cac317ef51331347746668a5aef57383003"}
Nov 03 18:09:12.667941 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:12.667821    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e44720ae1ae1d75575e6cc8881d44cac317ef51331347746668a5aef57383003"
Nov 03 18:09:12.854930 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:12.854891    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-248f8ee3-576cfc4795-5q9nf"]
Nov 03 18:09:16.031006 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:16.030970    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67"]
Nov 03 18:09:16.031760 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:16.031743    5315 cpu_manager.go:414] "containerMap: removing container" podUID="885b7b64-07d4-4d24-b961-77841e1f4ba8" containerName="revoke-keys"
Nov 03 18:09:16.031826 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:16.031760    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="885b7b64-07d4-4d24-b961-77841e1f4ba8" containerName="revoke-keys"
Nov 03 18:09:16.031901 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:16.031892    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="885b7b64-07d4-4d24-b961-77841e1f4ba8" containerName="revoke-keys"
Nov 03 18:09:16.034662 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:16.034599    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67"
Nov 03 18:09:16.060871 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:16.060844    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7899300-736f-4a03-aef7-253d7a1b1e2f-istiod-ca-cert\") pod \"e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67\" (UID: \"d7899300-736f-4a03-aef7-253d7a1b1e2f\") " pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67"
Nov 03 18:09:16.061005 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:16.060914    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7899300-736f-4a03-aef7-253d7a1b1e2f-istio-data\") pod \"e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67\" (UID: \"d7899300-736f-4a03-aef7-253d7a1b1e2f\") " pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67"
Nov 03 18:09:16.061048 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:16.061008    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7899300-736f-4a03-aef7-253d7a1b1e2f-istio-envoy\") pod \"e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67\" (UID: \"d7899300-736f-4a03-aef7-253d7a1b1e2f\") " pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67"
Nov 03 18:09:16.061048 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:16.061038    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7899300-736f-4a03-aef7-253d7a1b1e2f-istio-token\") pod \"e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67\" (UID: \"d7899300-736f-4a03-aef7-253d7a1b1e2f\") " pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67"
Nov 03 18:09:16.061163 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:16.061141    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2v2j\" (UniqueName: \"kubernetes.io/projected/d7899300-736f-4a03-aef7-253d7a1b1e2f-kube-api-access-b2v2j\") pod \"e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67\" (UID: \"d7899300-736f-4a03-aef7-253d7a1b1e2f\") " pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67"
Nov 03 18:09:16.061249 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:16.061236    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7899300-736f-4a03-aef7-253d7a1b1e2f-credential-socket\") pod \"e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67\" (UID: \"d7899300-736f-4a03-aef7-253d7a1b1e2f\") " pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67"
Nov 03 18:09:16.061300 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:16.061289    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7899300-736f-4a03-aef7-253d7a1b1e2f-workload-socket\") pod \"e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67\" (UID: \"d7899300-736f-4a03-aef7-253d7a1b1e2f\") " pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67"
Nov 03 18:09:16.061335 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:16.061310    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7899300-736f-4a03-aef7-253d7a1b1e2f-workload-certs\") pod \"e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67\" (UID: \"d7899300-736f-4a03-aef7-253d7a1b1e2f\") " pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67"
Nov 03 18:09:16.061414 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:16.061392    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7899300-736f-4a03-aef7-253d7a1b1e2f-istio-podinfo\") pod \"e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67\" (UID: \"d7899300-736f-4a03-aef7-253d7a1b1e2f\") " pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67"
Nov 03 18:09:16.090087 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:16.090052    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-c91bee-openshift-default-dockercfg-26bjc\""
Nov 03 18:09:16.144544 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:16.144511    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67"]
Nov 03 18:09:16.170644 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:16.170583    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7899300-736f-4a03-aef7-253d7a1b1e2f-istio-envoy\") pod \"e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67\" (UID: \"d7899300-736f-4a03-aef7-253d7a1b1e2f\") " pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67"
Nov 03 18:09:16.171435 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:16.171407    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7899300-736f-4a03-aef7-253d7a1b1e2f-istio-token\") pod \"e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67\" (UID: \"d7899300-736f-4a03-aef7-253d7a1b1e2f\") " pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67"
Nov 03 18:09:16.172075 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:16.172050    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2v2j\" (UniqueName: \"kubernetes.io/projected/d7899300-736f-4a03-aef7-253d7a1b1e2f-kube-api-access-b2v2j\") pod \"e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67\" (UID: \"d7899300-736f-4a03-aef7-253d7a1b1e2f\") " pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67"
Nov 03 18:09:16.172573 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:16.172549    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7899300-736f-4a03-aef7-253d7a1b1e2f-credential-socket\") pod \"e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67\" (UID: \"d7899300-736f-4a03-aef7-253d7a1b1e2f\") " pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67"
Nov 03 18:09:16.173210 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:16.173189    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7899300-736f-4a03-aef7-253d7a1b1e2f-workload-socket\") pod \"e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67\" (UID: \"d7899300-736f-4a03-aef7-253d7a1b1e2f\") " pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67"
Nov 03 18:09:16.173671 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:16.173651    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7899300-736f-4a03-aef7-253d7a1b1e2f-workload-certs\") pod \"e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67\" (UID: \"d7899300-736f-4a03-aef7-253d7a1b1e2f\") " pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67"
Nov 03 18:09:16.173902 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:16.173882    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7899300-736f-4a03-aef7-253d7a1b1e2f-istio-podinfo\") pod \"e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67\" (UID: \"d7899300-736f-4a03-aef7-253d7a1b1e2f\") " pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67"
Nov 03 18:09:16.174912 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:16.174889    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7899300-736f-4a03-aef7-253d7a1b1e2f-istiod-ca-cert\") pod \"e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67\" (UID: \"d7899300-736f-4a03-aef7-253d7a1b1e2f\") " pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67"
Nov 03 18:09:16.176308 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:16.173011    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7899300-736f-4a03-aef7-253d7a1b1e2f-credential-socket\") pod \"e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67\" (UID: \"d7899300-736f-4a03-aef7-253d7a1b1e2f\") " pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67"
Nov 03 18:09:16.176465 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:16.173589    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7899300-736f-4a03-aef7-253d7a1b1e2f-workload-socket\") pod \"e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67\" (UID: \"d7899300-736f-4a03-aef7-253d7a1b1e2f\") " pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67"
Nov 03 18:09:16.176570 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:16.174255    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7899300-736f-4a03-aef7-253d7a1b1e2f-workload-certs\") pod \"e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67\" (UID: \"d7899300-736f-4a03-aef7-253d7a1b1e2f\") " pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67"
Nov 03 18:09:16.183717 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:16.176186    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7899300-736f-4a03-aef7-253d7a1b1e2f-istiod-ca-cert\") pod \"e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67\" (UID: \"d7899300-736f-4a03-aef7-253d7a1b1e2f\") " pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67"
Nov 03 18:09:16.341334 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:16.268991    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7899300-736f-4a03-aef7-253d7a1b1e2f-istio-data\") pod \"e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67\" (UID: \"d7899300-736f-4a03-aef7-253d7a1b1e2f\") " pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67"
Nov 03 18:09:16.341334 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:16.269572    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7899300-736f-4a03-aef7-253d7a1b1e2f-istio-data\") pod \"e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67\" (UID: \"d7899300-736f-4a03-aef7-253d7a1b1e2f\") " pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67"
Nov 03 18:09:16.341334 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:16.302440    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7899300-736f-4a03-aef7-253d7a1b1e2f-istio-token\") pod \"e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67\" (UID: \"d7899300-736f-4a03-aef7-253d7a1b1e2f\") " pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67"
Nov 03 18:09:16.341334 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:16.317738    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7899300-736f-4a03-aef7-253d7a1b1e2f-istio-podinfo\") pod \"e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67\" (UID: \"d7899300-736f-4a03-aef7-253d7a1b1e2f\") " pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67"
Nov 03 18:09:16.381972 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:16.376863    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2v2j\" (UniqueName: \"kubernetes.io/projected/d7899300-736f-4a03-aef7-253d7a1b1e2f-kube-api-access-b2v2j\") pod \"e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67\" (UID: \"d7899300-736f-4a03-aef7-253d7a1b1e2f\") " pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67"
Nov 03 18:09:16.381972 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:16.377224    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7899300-736f-4a03-aef7-253d7a1b1e2f-istio-envoy\") pod \"e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67\" (UID: \"d7899300-736f-4a03-aef7-253d7a1b1e2f\") " pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67"
Nov 03 18:09:16.518356 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:16.510867    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-86101f5d-6ffcj"]
Nov 03 18:09:16.562424 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:16.562351    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-86101f5d-6ffcj"]
Nov 03 18:09:16.572231 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:16.572201    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns"]
Nov 03 18:09:16.572687 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:16.572638    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns" podUID="8832b3d8-1880-4277-8a0c-f9bf94eef7b3" containerName="istio-proxy" containerID="cri-o://8b456c26e6f2ebf28ceb6430fbb16922ede514eda2e46a44d11cd169622edf3c" gracePeriod=30
Nov 03 18:09:16.666922 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:16.659208    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67"
Nov 03 18:09:17.107467 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:17.107427    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b64f8fe2-0b09-47d9-981c-3dd32f24169f" path="/var/lib/kubelet/pods/b64f8fe2-0b09-47d9-981c-3dd32f24169f/volumes"
Nov 03 18:09:17.434528 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:17.434492    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67"]
Nov 03 18:09:17.437088 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:09:17.437058    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7899300_736f_4a03_aef7_253d7a1b1e2f.slice/crio-73e94ca374c06fe77e918bd7d4422d2f782937f3507b954b29b016ef43152e6d WatchSource:0}: Error finding container 73e94ca374c06fe77e918bd7d4422d2f782937f3507b954b29b016ef43152e6d: Status 404 returned error can't find the container with id 73e94ca374c06fe77e918bd7d4422d2f782937f3507b954b29b016ef43152e6d
Nov 03 18:09:17.440639 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:17.440581    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:09:17.440730 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:17.440691    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:09:17.440767 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:17.440734    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:09:17.657371 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:17.657331    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-248f8ee3-574c4c5b5-rqjv5" podUID="1ced2f03-d1ad-4a25-aa71-d6ece3178be9" containerName="payload-processing" containerID="cri-o://db8941ae2d8da604976cbe28d6e9658d8875a2106d9612ed4356380a82ece702" gracePeriod=25
Nov 03 18:09:17.856266 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:17.856226    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-248f8ee3-576cfc4795-5q9nf" podUID="05885de3-0afb-4ae7-a7d2-7c3b64e0bba1" containerName="payload-pre-processing" containerID="cri-o://7862d2ded6fe9289c25dd8dfc0976c69c0083a717ce64057757161dfa64489a1" gracePeriod=25
Nov 03 18:09:17.883242 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:17.883208    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c"]
Nov 03 18:09:17.893283 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:17.893230    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c"
Nov 03 18:09:18.270275 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.270183    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-5ffd208e-openshift-default-dockercfg-8sgb7\""
Nov 03 18:09:18.311142 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.311111    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c"]
Nov 03 18:09:18.314670 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.312439    5315 patch_prober.go:29] interesting pod/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:09:18.314670 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.312911    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns" podUID="8832b3d8-1880-4277-8a0c-f9bf94eef7b3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:09:18.337784 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.337747    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38acc846-2251-4000-ae98-e5a2492da6a6-credential-socket\") pod \"e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c\" (UID: \"38acc846-2251-4000-ae98-e5a2492da6a6\") " pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c"
Nov 03 18:09:18.343237 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.343203    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38acc846-2251-4000-ae98-e5a2492da6a6-istio-podinfo\") pod \"e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c\" (UID: \"38acc846-2251-4000-ae98-e5a2492da6a6\") " pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c"
Nov 03 18:09:18.343603 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.343576    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38acc846-2251-4000-ae98-e5a2492da6a6-istio-token\") pod \"e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c\" (UID: \"38acc846-2251-4000-ae98-e5a2492da6a6\") " pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c"
Nov 03 18:09:18.343880 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.343860    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38acc846-2251-4000-ae98-e5a2492da6a6-istiod-ca-cert\") pod \"e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c\" (UID: \"38acc846-2251-4000-ae98-e5a2492da6a6\") " pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c"
Nov 03 18:09:18.344070 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.344044    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38acc846-2251-4000-ae98-e5a2492da6a6-workload-certs\") pod \"e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c\" (UID: \"38acc846-2251-4000-ae98-e5a2492da6a6\") " pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c"
Nov 03 18:09:18.344838 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.344818    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hdxx\" (UniqueName: \"kubernetes.io/projected/38acc846-2251-4000-ae98-e5a2492da6a6-kube-api-access-7hdxx\") pod \"e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c\" (UID: \"38acc846-2251-4000-ae98-e5a2492da6a6\") " pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c"
Nov 03 18:09:18.364213 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.364166    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38acc846-2251-4000-ae98-e5a2492da6a6-istio-data\") pod \"e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c\" (UID: \"38acc846-2251-4000-ae98-e5a2492da6a6\") " pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c"
Nov 03 18:09:18.364539 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.364511    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38acc846-2251-4000-ae98-e5a2492da6a6-istio-envoy\") pod \"e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c\" (UID: \"38acc846-2251-4000-ae98-e5a2492da6a6\") " pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c"
Nov 03 18:09:18.364777 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.364755    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38acc846-2251-4000-ae98-e5a2492da6a6-workload-socket\") pod \"e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c\" (UID: \"38acc846-2251-4000-ae98-e5a2492da6a6\") " pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c"
Nov 03 18:09:18.396403 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.396364    5315 generic.go:356] "Generic (PLEG): container finished" podID="1ced2f03-d1ad-4a25-aa71-d6ece3178be9" containerID="db8941ae2d8da604976cbe28d6e9658d8875a2106d9612ed4356380a82ece702" exitCode=0
Nov 03 18:09:18.398364 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.398326    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-248f8ee3-574c4c5b5-rqjv5" event={"ID":"1ced2f03-d1ad-4a25-aa71-d6ece3178be9","Type":"ContainerDied","Data":"db8941ae2d8da604976cbe28d6e9658d8875a2106d9612ed4356380a82ece702"}
Nov 03 18:09:18.398646 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.398624    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-248f8ee3-574c4c5b5-rqjv5" event={"ID":"1ced2f03-d1ad-4a25-aa71-d6ece3178be9","Type":"ContainerDied","Data":"84b91f01b9a96437f84a8d6b0dbfb015c4dd8d792bcdcaae0b517c4bf10c5b50"}
Nov 03 18:09:18.398783 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.398768    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84b91f01b9a96437f84a8d6b0dbfb015c4dd8d792bcdcaae0b517c4bf10c5b50"
Nov 03 18:09:18.455133 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.455091    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67" event={"ID":"d7899300-736f-4a03-aef7-253d7a1b1e2f","Type":"ContainerStarted","Data":"750e09020663ab70c545beeced3a22c7d61ce9e5431603a6d98aa87a8eeb4264"}
Nov 03 18:09:18.455416 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.455394    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67" event={"ID":"d7899300-736f-4a03-aef7-253d7a1b1e2f","Type":"ContainerStarted","Data":"73e94ca374c06fe77e918bd7d4422d2f782937f3507b954b29b016ef43152e6d"}
Nov 03 18:09:18.484720 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.482181    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hdxx\" (UniqueName: \"kubernetes.io/projected/38acc846-2251-4000-ae98-e5a2492da6a6-kube-api-access-7hdxx\") pod \"e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c\" (UID: \"38acc846-2251-4000-ae98-e5a2492da6a6\") " pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c"
Nov 03 18:09:18.484720 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.482266    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38acc846-2251-4000-ae98-e5a2492da6a6-istio-data\") pod \"e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c\" (UID: \"38acc846-2251-4000-ae98-e5a2492da6a6\") " pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c"
Nov 03 18:09:18.484720 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.482329    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38acc846-2251-4000-ae98-e5a2492da6a6-istio-envoy\") pod \"e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c\" (UID: \"38acc846-2251-4000-ae98-e5a2492da6a6\") " pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c"
Nov 03 18:09:18.484720 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.482416    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38acc846-2251-4000-ae98-e5a2492da6a6-workload-socket\") pod \"e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c\" (UID: \"38acc846-2251-4000-ae98-e5a2492da6a6\") " pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c"
Nov 03 18:09:18.484720 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.482619    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38acc846-2251-4000-ae98-e5a2492da6a6-credential-socket\") pod \"e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c\" (UID: \"38acc846-2251-4000-ae98-e5a2492da6a6\") " pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c"
Nov 03 18:09:18.484720 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.482652    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38acc846-2251-4000-ae98-e5a2492da6a6-istio-podinfo\") pod \"e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c\" (UID: \"38acc846-2251-4000-ae98-e5a2492da6a6\") " pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c"
Nov 03 18:09:18.484720 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.482720    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38acc846-2251-4000-ae98-e5a2492da6a6-istio-token\") pod \"e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c\" (UID: \"38acc846-2251-4000-ae98-e5a2492da6a6\") " pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c"
Nov 03 18:09:18.484720 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.482823    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38acc846-2251-4000-ae98-e5a2492da6a6-istiod-ca-cert\") pod \"e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c\" (UID: \"38acc846-2251-4000-ae98-e5a2492da6a6\") " pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c"
Nov 03 18:09:18.484720 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.482892    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38acc846-2251-4000-ae98-e5a2492da6a6-workload-certs\") pod \"e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c\" (UID: \"38acc846-2251-4000-ae98-e5a2492da6a6\") " pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c"
Nov 03 18:09:18.484720 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.483396    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38acc846-2251-4000-ae98-e5a2492da6a6-workload-certs\") pod \"e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c\" (UID: \"38acc846-2251-4000-ae98-e5a2492da6a6\") " pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c"
Nov 03 18:09:18.525595 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.509505    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38acc846-2251-4000-ae98-e5a2492da6a6-istio-data\") pod \"e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c\" (UID: \"38acc846-2251-4000-ae98-e5a2492da6a6\") " pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c"
Nov 03 18:09:18.525595 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.518128    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38acc846-2251-4000-ae98-e5a2492da6a6-istio-envoy\") pod \"e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c\" (UID: \"38acc846-2251-4000-ae98-e5a2492da6a6\") " pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c"
Nov 03 18:09:18.525595 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.518342    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38acc846-2251-4000-ae98-e5a2492da6a6-workload-socket\") pod \"e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c\" (UID: \"38acc846-2251-4000-ae98-e5a2492da6a6\") " pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c"
Nov 03 18:09:18.525595 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.518512    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38acc846-2251-4000-ae98-e5a2492da6a6-credential-socket\") pod \"e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c\" (UID: \"38acc846-2251-4000-ae98-e5a2492da6a6\") " pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c"
Nov 03 18:09:18.530196 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.529762    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38acc846-2251-4000-ae98-e5a2492da6a6-istiod-ca-cert\") pod \"e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c\" (UID: \"38acc846-2251-4000-ae98-e5a2492da6a6\") " pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c"
Nov 03 18:09:18.531004 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.530845    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-248f8ee3-574c4c5b5-rqjv5"
Nov 03 18:09:18.602249 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.585810    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ced2f03-d1ad-4a25-aa71-d6ece3178be9-kube-api-access-7sff2\" (UniqueName: \"kubernetes.io/projected/1ced2f03-d1ad-4a25-aa71-d6ece3178be9-kube-api-access-7sff2\") pod \"1ced2f03-d1ad-4a25-aa71-d6ece3178be9\" (UID: \"1ced2f03-d1ad-4a25-aa71-d6ece3178be9\") "
Nov 03 18:09:18.602249 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.586063    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ced2f03-d1ad-4a25-aa71-d6ece3178be9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ced2f03-d1ad-4a25-aa71-d6ece3178be9-plugins-config-volume\") pod \"1ced2f03-d1ad-4a25-aa71-d6ece3178be9\" (UID: \"1ced2f03-d1ad-4a25-aa71-d6ece3178be9\") "
Nov 03 18:09:18.622798 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.621368    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ced2f03-d1ad-4a25-aa71-d6ece3178be9-plugins-config-volume" pod "1ced2f03-d1ad-4a25-aa71-d6ece3178be9" (UID: "1ced2f03-d1ad-4a25-aa71-d6ece3178be9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:09:18.680994 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.672250    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38acc846-2251-4000-ae98-e5a2492da6a6-istio-token\") pod \"e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c\" (UID: \"38acc846-2251-4000-ae98-e5a2492da6a6\") " pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c"
Nov 03 18:09:18.680994 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.672419    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ced2f03-d1ad-4a25-aa71-d6ece3178be9-kube-api-access-7sff2" pod "1ced2f03-d1ad-4a25-aa71-d6ece3178be9" (UID: "1ced2f03-d1ad-4a25-aa71-d6ece3178be9"). InnerVolumeSpecName "kube-api-access-7sff2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:09:18.680994 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.673050    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38acc846-2251-4000-ae98-e5a2492da6a6-istio-podinfo\") pod \"e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c\" (UID: \"38acc846-2251-4000-ae98-e5a2492da6a6\") " pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c"
Nov 03 18:09:18.680994 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.676430    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67"
Nov 03 18:09:18.680994 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.677143    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hdxx\" (UniqueName: \"kubernetes.io/projected/38acc846-2251-4000-ae98-e5a2492da6a6-kube-api-access-7hdxx\") pod \"e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c\" (UID: \"38acc846-2251-4000-ae98-e5a2492da6a6\") " pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c"
Nov 03 18:09:18.704177 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.704133    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c"
Nov 03 18:09:18.704914 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.704886    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sff2\" (UniqueName: \"kubernetes.io/projected/1ced2f03-d1ad-4a25-aa71-d6ece3178be9-kube-api-access-7sff2\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:18.705202 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.705176    5315 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ced2f03-d1ad-4a25-aa71-d6ece3178be9-plugins-config-volume\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:18.742880 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.742812    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67" podStartSLOduration=3.742604828 podStartE2EDuration="3.742604828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:09:18.64033267 +0000 UTC m=+1397.656530739" watchObservedRunningTime="2026-11-03 18:09:18.742604828 +0000 UTC m=+1397.758802898"
Nov 03 18:09:18.751811 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.751779    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67"
Nov 03 18:09:18.983523 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.983499    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-248f8ee3-576cfc4795-5q9nf"
Nov 03 18:09:18.997422 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.997384    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05885de3-0afb-4ae7-a7d2-7c3b64e0bba1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05885de3-0afb-4ae7-a7d2-7c3b64e0bba1-plugins-config-volume\") pod \"05885de3-0afb-4ae7-a7d2-7c3b64e0bba1\" (UID: \"05885de3-0afb-4ae7-a7d2-7c3b64e0bba1\") "
Nov 03 18:09:18.997765 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.997739    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05885de3-0afb-4ae7-a7d2-7c3b64e0bba1-kube-api-access-j7j6q\" (UniqueName: \"kubernetes.io/projected/05885de3-0afb-4ae7-a7d2-7c3b64e0bba1-kube-api-access-j7j6q\") pod \"05885de3-0afb-4ae7-a7d2-7c3b64e0bba1\" (UID: \"05885de3-0afb-4ae7-a7d2-7c3b64e0bba1\") "
Nov 03 18:09:18.997869 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.997818    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05885de3-0afb-4ae7-a7d2-7c3b64e0bba1-plugins-config-volume" pod "05885de3-0afb-4ae7-a7d2-7c3b64e0bba1" (UID: "05885de3-0afb-4ae7-a7d2-7c3b64e0bba1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:09:18.998586 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:18.998557    5315 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05885de3-0afb-4ae7-a7d2-7c3b64e0bba1-plugins-config-volume\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:19.000097 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:19.000072    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05885de3-0afb-4ae7-a7d2-7c3b64e0bba1-kube-api-access-j7j6q" pod "05885de3-0afb-4ae7-a7d2-7c3b64e0bba1" (UID: "05885de3-0afb-4ae7-a7d2-7c3b64e0bba1"). InnerVolumeSpecName "kube-api-access-j7j6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:09:19.100603 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:19.100555    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7j6q\" (UniqueName: \"kubernetes.io/projected/05885de3-0afb-4ae7-a7d2-7c3b64e0bba1-kube-api-access-j7j6q\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:19.348892 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:19.348802    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c"]
Nov 03 18:09:19.351976 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:09:19.351944    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38acc846_2251_4000_ae98_e5a2492da6a6.slice/crio-8bcfaa5b49538f5ef285ea1f570a583104a19ebd5636498c326934a43551dc90 WatchSource:0}: Error finding container 8bcfaa5b49538f5ef285ea1f570a583104a19ebd5636498c326934a43551dc90: Status 404 returned error can't find the container with id 8bcfaa5b49538f5ef285ea1f570a583104a19ebd5636498c326934a43551dc90
Nov 03 18:09:19.354882 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:19.354851    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:09:19.354961 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:19.354925    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:09:19.354961 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:19.354957    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:09:19.466767 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:19.466722    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c" event={"ID":"38acc846-2251-4000-ae98-e5a2492da6a6","Type":"ContainerStarted","Data":"4f68f393ea3076a8de212d5b8c368a7c2cfd504c1693f3bbf8e150c268a96686"}
Nov 03 18:09:19.466964 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:19.466802    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c" event={"ID":"38acc846-2251-4000-ae98-e5a2492da6a6","Type":"ContainerStarted","Data":"8bcfaa5b49538f5ef285ea1f570a583104a19ebd5636498c326934a43551dc90"}
Nov 03 18:09:19.468447 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:19.468422    5315 generic.go:356] "Generic (PLEG): container finished" podID="05885de3-0afb-4ae7-a7d2-7c3b64e0bba1" containerID="7862d2ded6fe9289c25dd8dfc0976c69c0083a717ce64057757161dfa64489a1" exitCode=0
Nov 03 18:09:19.468583 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:19.468508    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-248f8ee3-576cfc4795-5q9nf"
Nov 03 18:09:19.468583 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:19.468541    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-248f8ee3-576cfc4795-5q9nf" event={"ID":"05885de3-0afb-4ae7-a7d2-7c3b64e0bba1","Type":"ContainerDied","Data":"7862d2ded6fe9289c25dd8dfc0976c69c0083a717ce64057757161dfa64489a1"}
Nov 03 18:09:19.468711 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:19.468598    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-248f8ee3-576cfc4795-5q9nf" event={"ID":"05885de3-0afb-4ae7-a7d2-7c3b64e0bba1","Type":"ContainerDied","Data":"dceef1823463c0aee0af310cd70c06e9d1193080b8e7bbcc0e582979e8325762"}
Nov 03 18:09:19.468711 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:19.468666    5315 scope.go:122] "RemoveContainer" containerID="7862d2ded6fe9289c25dd8dfc0976c69c0083a717ce64057757161dfa64489a1"
Nov 03 18:09:19.468813 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:19.468778    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-248f8ee3-574c4c5b5-rqjv5"
Nov 03 18:09:19.469560 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:19.469514    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67"
Nov 03 18:09:19.471072 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:19.470757    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67"
Nov 03 18:09:19.481681 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:19.481659    5315 scope.go:122] "RemoveContainer" containerID="7862d2ded6fe9289c25dd8dfc0976c69c0083a717ce64057757161dfa64489a1"
Nov 03 18:09:19.482006 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:09:19.481979    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7862d2ded6fe9289c25dd8dfc0976c69c0083a717ce64057757161dfa64489a1\": container with ID starting with 7862d2ded6fe9289c25dd8dfc0976c69c0083a717ce64057757161dfa64489a1 not found: ID does not exist" containerID="7862d2ded6fe9289c25dd8dfc0976c69c0083a717ce64057757161dfa64489a1"
Nov 03 18:09:19.482083 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:19.482015    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7862d2ded6fe9289c25dd8dfc0976c69c0083a717ce64057757161dfa64489a1"} err="failed to get container status \"7862d2ded6fe9289c25dd8dfc0976c69c0083a717ce64057757161dfa64489a1\": rpc error: code = NotFound desc = could not find container \"7862d2ded6fe9289c25dd8dfc0976c69c0083a717ce64057757161dfa64489a1\": container with ID starting with 7862d2ded6fe9289c25dd8dfc0976c69c0083a717ce64057757161dfa64489a1 not found: ID does not exist"
Nov 03 18:09:19.580568 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:19.580461    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c" podStartSLOduration=2.5804470200000003 podStartE2EDuration="2.58044702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:09:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:09:19.579219498 +0000 UTC m=+1398.595417568" watchObservedRunningTime="2026-11-03 18:09:19.58044702 +0000 UTC m=+1398.596645090"
Nov 03 18:09:19.687663 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:19.687629    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-248f8ee3-576cfc4795-5q9nf"]
Nov 03 18:09:19.693426 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:19.693393    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-248f8ee3-576cfc4795-5q9nf"]
Nov 03 18:09:19.743446 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:19.716703    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c"
Nov 03 18:09:19.855693 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:19.851166    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4b6e7259-54b9769cbc-k5qbr"]
Nov 03 18:09:19.911144 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:19.911112    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4b6e7259-854d546fff-j5qbr"]
Nov 03 18:09:19.933553 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:19.933526    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-248f8ee3-574c4c5b5-rqjv5"]
Nov 03 18:09:19.940507 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:19.940477    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-248f8ee3-574c4c5b5-rqjv5"]
Nov 03 18:09:20.776527 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:20.774531    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c"
Nov 03 18:09:21.150420 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:21.150320    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05885de3-0afb-4ae7-a7d2-7c3b64e0bba1" path="/var/lib/kubelet/pods/05885de3-0afb-4ae7-a7d2-7c3b64e0bba1/volumes"
Nov 03 18:09:21.151198 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:21.151166    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ced2f03-d1ad-4a25-aa71-d6ece3178be9" path="/var/lib/kubelet/pods/1ced2f03-d1ad-4a25-aa71-d6ece3178be9/volumes"
Nov 03 18:09:21.731803 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:21.731765    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c"
Nov 03 18:09:21.741129 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:21.741097    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c"
Nov 03 18:09:22.737237 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:22.737194    5315 generic.go:356] "Generic (PLEG): container finished" podID="8832b3d8-1880-4277-8a0c-f9bf94eef7b3" containerID="8b456c26e6f2ebf28ceb6430fbb16922ede514eda2e46a44d11cd169622edf3c" exitCode=0
Nov 03 18:09:22.737730 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:22.737245    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns" event={"ID":"8832b3d8-1880-4277-8a0c-f9bf94eef7b3","Type":"ContainerDied","Data":"8b456c26e6f2ebf28ceb6430fbb16922ede514eda2e46a44d11cd169622edf3c"}
Nov 03 18:09:22.737730 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:22.737369    5315 scope.go:122] "RemoveContainer" containerID="767d0ab15d11016a2d5140fad23874f7304cd7ecc9136f36983a08713a478858"
Nov 03 18:09:22.861164 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:22.861139    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns"
Nov 03 18:09:22.882930 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:22.882901    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-29bb4e29-42cgh"]
Nov 03 18:09:22.891715 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:22.891689    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-29bb4e29-42cgh"]
Nov 03 18:09:23.027545 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:23.027511    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-istio-token\" (UniqueName: \"kubernetes.io/projected/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-istio-token\") pod \"8832b3d8-1880-4277-8a0c-f9bf94eef7b3\" (UID: \"8832b3d8-1880-4277-8a0c-f9bf94eef7b3\") "
Nov 03 18:09:23.027716 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:23.027666    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-workload-certs\") pod \"8832b3d8-1880-4277-8a0c-f9bf94eef7b3\" (UID: \"8832b3d8-1880-4277-8a0c-f9bf94eef7b3\") "
Nov 03 18:09:23.027716 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:23.027706    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-credential-socket\") pod \"8832b3d8-1880-4277-8a0c-f9bf94eef7b3\" (UID: \"8832b3d8-1880-4277-8a0c-f9bf94eef7b3\") "
Nov 03 18:09:23.027854 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:23.027768    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-istio-data\") pod \"8832b3d8-1880-4277-8a0c-f9bf94eef7b3\" (UID: \"8832b3d8-1880-4277-8a0c-f9bf94eef7b3\") "
Nov 03 18:09:23.027854 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:23.027840    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-workload-socket\") pod \"8832b3d8-1880-4277-8a0c-f9bf94eef7b3\" (UID: \"8832b3d8-1880-4277-8a0c-f9bf94eef7b3\") "
Nov 03 18:09:23.028037 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:23.028008    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-workload-certs" pod "8832b3d8-1880-4277-8a0c-f9bf94eef7b3" (UID: "8832b3d8-1880-4277-8a0c-f9bf94eef7b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:23.028117 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:23.028075    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-istio-data" pod "8832b3d8-1880-4277-8a0c-f9bf94eef7b3" (UID: "8832b3d8-1880-4277-8a0c-f9bf94eef7b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:23.028156 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:23.028112    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-workload-socket" pod "8832b3d8-1880-4277-8a0c-f9bf94eef7b3" (UID: "8832b3d8-1880-4277-8a0c-f9bf94eef7b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:23.028281 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:23.028233    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-istio-envoy\") pod \"8832b3d8-1880-4277-8a0c-f9bf94eef7b3\" (UID: \"8832b3d8-1880-4277-8a0c-f9bf94eef7b3\") "
Nov 03 18:09:23.028340 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:23.028298    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-credential-socket" pod "8832b3d8-1880-4277-8a0c-f9bf94eef7b3" (UID: "8832b3d8-1880-4277-8a0c-f9bf94eef7b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:23.028422 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:23.028394    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-istiod-ca-cert\") pod \"8832b3d8-1880-4277-8a0c-f9bf94eef7b3\" (UID: \"8832b3d8-1880-4277-8a0c-f9bf94eef7b3\") "
Nov 03 18:09:23.028485 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:23.028468    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-kube-api-access-n4zxz\" (UniqueName: \"kubernetes.io/projected/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-kube-api-access-n4zxz\") pod \"8832b3d8-1880-4277-8a0c-f9bf94eef7b3\" (UID: \"8832b3d8-1880-4277-8a0c-f9bf94eef7b3\") "
Nov 03 18:09:23.028557 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:23.028535    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-istio-podinfo\") pod \"8832b3d8-1880-4277-8a0c-f9bf94eef7b3\" (UID: \"8832b3d8-1880-4277-8a0c-f9bf94eef7b3\") "
Nov 03 18:09:23.028870 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:23.028845    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-istiod-ca-cert" pod "8832b3d8-1880-4277-8a0c-f9bf94eef7b3" (UID: "8832b3d8-1880-4277-8a0c-f9bf94eef7b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:09:23.029310 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:23.029277    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:23.029421 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:23.029310    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:23.029421 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:23.029330    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:23.029421 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:23.029344    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:23.029421 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:23.029357    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:23.030296 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:23.030269    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-istio-token" pod "8832b3d8-1880-4277-8a0c-f9bf94eef7b3" (UID: "8832b3d8-1880-4277-8a0c-f9bf94eef7b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:09:23.031106 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:23.031085    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-kube-api-access-n4zxz" pod "8832b3d8-1880-4277-8a0c-f9bf94eef7b3" (UID: "8832b3d8-1880-4277-8a0c-f9bf94eef7b3"). InnerVolumeSpecName "kube-api-access-n4zxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:09:23.031182 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:23.031141    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-istio-envoy" pod "8832b3d8-1880-4277-8a0c-f9bf94eef7b3" (UID: "8832b3d8-1880-4277-8a0c-f9bf94eef7b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:23.031482 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:23.031454    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-istio-podinfo" pod "8832b3d8-1880-4277-8a0c-f9bf94eef7b3" (UID: "8832b3d8-1880-4277-8a0c-f9bf94eef7b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:09:23.108953 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:23.108907    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d05f048d-1346-491b-bc11-121a6c7d53a6" path="/var/lib/kubelet/pods/d05f048d-1346-491b-bc11-121a6c7d53a6/volumes"
Nov 03 18:09:23.130528 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:23.130488    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4zxz\" (UniqueName: \"kubernetes.io/projected/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-kube-api-access-n4zxz\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:23.130528 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:23.130528    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:23.130528 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:23.130538    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:23.130752 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:23.130547    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8832b3d8-1880-4277-8a0c-f9bf94eef7b3-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:23.575452 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:23.575253    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt"]
Nov 03 18:09:23.575654 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:23.575542    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt" podUID="feb037bd-264e-486b-aa7d-f3679b439981" containerName="istio-proxy" containerID="cri-o://51097ba4de28cb7a57daccd00acaff578898ba5cf1f9164d51ed3177999e2ed7" gracePeriod=30
Nov 03 18:09:23.601767 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:23.601732    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg"]
Nov 03 18:09:23.601952 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:23.601917    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg" podUID="86e80b99-3e6f-41ae-8832-9cfbbf2e543a" containerName="istio-proxy" containerID="cri-o://0dfdd1466f5ab8b88a3e96c4d98823ffdbe87503efd084a453c49b0d9a80e2cf" gracePeriod=30
Nov 03 18:09:23.903831 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:23.901543    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns"
Nov 03 18:09:23.903831 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:23.901573    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns" event={"ID":"8832b3d8-1880-4277-8a0c-f9bf94eef7b3","Type":"ContainerDied","Data":"42072296ac0af62669dda6e644897cc9961e262f2a0c97c796549471401d2c5e"}
Nov 03 18:09:23.903831 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:23.901656    5315 scope.go:122] "RemoveContainer" containerID="8b456c26e6f2ebf28ceb6430fbb16922ede514eda2e46a44d11cd169622edf3c"
Nov 03 18:09:24.067643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:24.067601    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns"]
Nov 03 18:09:24.072970 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:24.072943    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-86101f5d-gw-openshift-default-d5d96f665-7xlns"]
Nov 03 18:09:24.582839 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:09:24.582798    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:09:24.852510 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:24.852421    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-4b6e7259-54b9769cbc-k5qbr" podUID="fdafb581-457b-4a16-a9b7-9c19f90f30cd" containerName="payload-processing" containerID="cri-o://543ad04eb72d611013219d1ef3907b5a3998c949737db9f8458c8243d92ed4fc" gracePeriod=25
Nov 03 18:09:24.911449 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:24.911411    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b6e7259-854d546fff-j5qbr" podUID="1dd7528b-e815-44b0-aa17-72f16301468e" containerName="payload-pre-processing" containerID="cri-o://270400b33a1623b292a8a153d05f6c79d21632ded1f82c8fba0f00ba3e356520" gracePeriod=25
Nov 03 18:09:25.112344 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:25.112264    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8832b3d8-1880-4277-8a0c-f9bf94eef7b3" path="/var/lib/kubelet/pods/8832b3d8-1880-4277-8a0c-f9bf94eef7b3/volumes"
Nov 03 18:09:25.204163 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:25.203912    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4b6e7259-54b9769cbc-k5qbr"
Nov 03 18:09:25.255256 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:25.255226    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b6e7259-854d546fff-j5qbr"
Nov 03 18:09:25.289346 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:25.289313    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdafb581-457b-4a16-a9b7-9c19f90f30cd-kube-api-access-rwhp2\" (UniqueName: \"kubernetes.io/projected/fdafb581-457b-4a16-a9b7-9c19f90f30cd-kube-api-access-rwhp2\") pod \"fdafb581-457b-4a16-a9b7-9c19f90f30cd\" (UID: \"fdafb581-457b-4a16-a9b7-9c19f90f30cd\") "
Nov 03 18:09:25.289583 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:25.289569    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdafb581-457b-4a16-a9b7-9c19f90f30cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdafb581-457b-4a16-a9b7-9c19f90f30cd-plugins-config-volume\") pod \"fdafb581-457b-4a16-a9b7-9c19f90f30cd\" (UID: \"fdafb581-457b-4a16-a9b7-9c19f90f30cd\") "
Nov 03 18:09:25.289897 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:25.289863    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdafb581-457b-4a16-a9b7-9c19f90f30cd-plugins-config-volume" pod "fdafb581-457b-4a16-a9b7-9c19f90f30cd" (UID: "fdafb581-457b-4a16-a9b7-9c19f90f30cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:09:25.290151 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:25.290136    5315 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdafb581-457b-4a16-a9b7-9c19f90f30cd-plugins-config-volume\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:25.291337 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:25.291314    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdafb581-457b-4a16-a9b7-9c19f90f30cd-kube-api-access-rwhp2" pod "fdafb581-457b-4a16-a9b7-9c19f90f30cd" (UID: "fdafb581-457b-4a16-a9b7-9c19f90f30cd"). InnerVolumeSpecName "kube-api-access-rwhp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:09:25.391529 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:25.391436    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dd7528b-e815-44b0-aa17-72f16301468e-kube-api-access-mrp28\" (UniqueName: \"kubernetes.io/projected/1dd7528b-e815-44b0-aa17-72f16301468e-kube-api-access-mrp28\") pod \"1dd7528b-e815-44b0-aa17-72f16301468e\" (UID: \"1dd7528b-e815-44b0-aa17-72f16301468e\") "
Nov 03 18:09:25.391529 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:25.391488    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dd7528b-e815-44b0-aa17-72f16301468e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dd7528b-e815-44b0-aa17-72f16301468e-plugins-config-volume\") pod \"1dd7528b-e815-44b0-aa17-72f16301468e\" (UID: \"1dd7528b-e815-44b0-aa17-72f16301468e\") "
Nov 03 18:09:25.391881 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:25.391860    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dd7528b-e815-44b0-aa17-72f16301468e-plugins-config-volume" pod "1dd7528b-e815-44b0-aa17-72f16301468e" (UID: "1dd7528b-e815-44b0-aa17-72f16301468e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:09:25.392061 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:25.392046    5315 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dd7528b-e815-44b0-aa17-72f16301468e-plugins-config-volume\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:25.392104 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:25.392064    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwhp2\" (UniqueName: \"kubernetes.io/projected/fdafb581-457b-4a16-a9b7-9c19f90f30cd-kube-api-access-rwhp2\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:25.393671 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:25.393647    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dd7528b-e815-44b0-aa17-72f16301468e-kube-api-access-mrp28" pod "1dd7528b-e815-44b0-aa17-72f16301468e" (UID: "1dd7528b-e815-44b0-aa17-72f16301468e"). InnerVolumeSpecName "kube-api-access-mrp28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:09:25.494018 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:25.493978    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrp28\" (UniqueName: \"kubernetes.io/projected/1dd7528b-e815-44b0-aa17-72f16301468e-kube-api-access-mrp28\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:26.010738 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:26.009879    5315 generic.go:356] "Generic (PLEG): container finished" podID="1dd7528b-e815-44b0-aa17-72f16301468e" containerID="270400b33a1623b292a8a153d05f6c79d21632ded1f82c8fba0f00ba3e356520" exitCode=0
Nov 03 18:09:26.010738 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:26.010002    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b6e7259-854d546fff-j5qbr" event={"ID":"1dd7528b-e815-44b0-aa17-72f16301468e","Type":"ContainerDied","Data":"270400b33a1623b292a8a153d05f6c79d21632ded1f82c8fba0f00ba3e356520"}
Nov 03 18:09:26.010738 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:26.010057    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b6e7259-854d546fff-j5qbr" event={"ID":"1dd7528b-e815-44b0-aa17-72f16301468e","Type":"ContainerDied","Data":"ecdcbcf6477d4107d221c10daa5b3a352cbcb4455437c8f00f8dc656bf6fe9a7"}
Nov 03 18:09:26.010738 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:26.010287    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b6e7259-854d546fff-j5qbr"
Nov 03 18:09:26.010738 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:26.010455    5315 scope.go:122] "RemoveContainer" containerID="270400b33a1623b292a8a153d05f6c79d21632ded1f82c8fba0f00ba3e356520"
Nov 03 18:09:26.018640 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:26.015242    5315 generic.go:356] "Generic (PLEG): container finished" podID="fdafb581-457b-4a16-a9b7-9c19f90f30cd" containerID="543ad04eb72d611013219d1ef3907b5a3998c949737db9f8458c8243d92ed4fc" exitCode=0
Nov 03 18:09:26.018640 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:26.015568    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4b6e7259-54b9769cbc-k5qbr" event={"ID":"fdafb581-457b-4a16-a9b7-9c19f90f30cd","Type":"ContainerDied","Data":"543ad04eb72d611013219d1ef3907b5a3998c949737db9f8458c8243d92ed4fc"}
Nov 03 18:09:26.018640 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:26.015637    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4b6e7259-54b9769cbc-k5qbr" event={"ID":"fdafb581-457b-4a16-a9b7-9c19f90f30cd","Type":"ContainerDied","Data":"85c69d2a5c307c07c63b3712a60118844b6cd3c130aa6d0f824bb4fc0001d27d"}
Nov 03 18:09:26.018640 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:26.015718    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4b6e7259-54b9769cbc-k5qbr"
Nov 03 18:09:26.077037 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:26.076742    5315 scope.go:122] "RemoveContainer" containerID="270400b33a1623b292a8a153d05f6c79d21632ded1f82c8fba0f00ba3e356520"
Nov 03 18:09:26.077292 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:09:26.077256    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"270400b33a1623b292a8a153d05f6c79d21632ded1f82c8fba0f00ba3e356520\": container with ID starting with 270400b33a1623b292a8a153d05f6c79d21632ded1f82c8fba0f00ba3e356520 not found: ID does not exist" containerID="270400b33a1623b292a8a153d05f6c79d21632ded1f82c8fba0f00ba3e356520"
Nov 03 18:09:26.077450 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:26.077423    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"270400b33a1623b292a8a153d05f6c79d21632ded1f82c8fba0f00ba3e356520"} err="failed to get container status \"270400b33a1623b292a8a153d05f6c79d21632ded1f82c8fba0f00ba3e356520\": rpc error: code = NotFound desc = could not find container \"270400b33a1623b292a8a153d05f6c79d21632ded1f82c8fba0f00ba3e356520\": container with ID starting with 270400b33a1623b292a8a153d05f6c79d21632ded1f82c8fba0f00ba3e356520 not found: ID does not exist"
Nov 03 18:09:26.077540 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:26.077529    5315 scope.go:122] "RemoveContainer" containerID="543ad04eb72d611013219d1ef3907b5a3998c949737db9f8458c8243d92ed4fc"
Nov 03 18:09:26.131542 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:26.130917    5315 scope.go:122] "RemoveContainer" containerID="543ad04eb72d611013219d1ef3907b5a3998c949737db9f8458c8243d92ed4fc"
Nov 03 18:09:26.132067 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:09:26.131941    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"543ad04eb72d611013219d1ef3907b5a3998c949737db9f8458c8243d92ed4fc\": container with ID starting with 543ad04eb72d611013219d1ef3907b5a3998c949737db9f8458c8243d92ed4fc not found: ID does not exist" containerID="543ad04eb72d611013219d1ef3907b5a3998c949737db9f8458c8243d92ed4fc"
Nov 03 18:09:26.132067 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:26.131987    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"543ad04eb72d611013219d1ef3907b5a3998c949737db9f8458c8243d92ed4fc"} err="failed to get container status \"543ad04eb72d611013219d1ef3907b5a3998c949737db9f8458c8243d92ed4fc\": rpc error: code = NotFound desc = could not find container \"543ad04eb72d611013219d1ef3907b5a3998c949737db9f8458c8243d92ed4fc\": container with ID starting with 543ad04eb72d611013219d1ef3907b5a3998c949737db9f8458c8243d92ed4fc not found: ID does not exist"
Nov 03 18:09:26.394836 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:26.394750    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4b6e7259-54b9769cbc-k5qbr"]
Nov 03 18:09:26.483021 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:26.482958    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4b6e7259-54b9769cbc-k5qbr"]
Nov 03 18:09:26.751321 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:26.751233    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-c91bee-76b997676c-zfjvq"]
Nov 03 18:09:26.752777 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:26.752744    5315 cpu_manager.go:414] "containerMap: removing container" podUID="fdafb581-457b-4a16-a9b7-9c19f90f30cd" containerName="payload-processing"
Nov 03 18:09:26.752891 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:26.752777    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdafb581-457b-4a16-a9b7-9c19f90f30cd" containerName="payload-processing"
Nov 03 18:09:26.752891 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:26.752842    5315 cpu_manager.go:414] "containerMap: removing container" podUID="8832b3d8-1880-4277-8a0c-f9bf94eef7b3" containerName="istio-proxy"
Nov 03 18:09:26.752891 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:26.752851    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8832b3d8-1880-4277-8a0c-f9bf94eef7b3" containerName="istio-proxy"
Nov 03 18:09:26.752891 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:26.752881    5315 cpu_manager.go:414] "containerMap: removing container" podUID="05885de3-0afb-4ae7-a7d2-7c3b64e0bba1" containerName="payload-pre-processing"
Nov 03 18:09:26.752891 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:26.752889    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05885de3-0afb-4ae7-a7d2-7c3b64e0bba1" containerName="payload-pre-processing"
Nov 03 18:09:26.753081 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:26.752924    5315 cpu_manager.go:414] "containerMap: removing container" podUID="1dd7528b-e815-44b0-aa17-72f16301468e" containerName="payload-pre-processing"
Nov 03 18:09:26.753081 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:26.752935    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dd7528b-e815-44b0-aa17-72f16301468e" containerName="payload-pre-processing"
Nov 03 18:09:26.753081 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:26.752950    5315 cpu_manager.go:414] "containerMap: removing container" podUID="1ced2f03-d1ad-4a25-aa71-d6ece3178be9" containerName="payload-processing"
Nov 03 18:09:26.753081 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:26.752957    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ced2f03-d1ad-4a25-aa71-d6ece3178be9" containerName="payload-processing"
Nov 03 18:09:26.780706 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:26.780675    5315 cpu_manager.go:414] "containerMap: removing container" podUID="8832b3d8-1880-4277-8a0c-f9bf94eef7b3" containerName="istio-proxy"
Nov 03 18:09:26.780904 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:26.780889    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8832b3d8-1880-4277-8a0c-f9bf94eef7b3" containerName="istio-proxy"
Nov 03 18:09:26.781243 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:26.781210    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="8832b3d8-1880-4277-8a0c-f9bf94eef7b3" containerName="istio-proxy"
Nov 03 18:09:26.781381 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:26.781370    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdafb581-457b-4a16-a9b7-9c19f90f30cd" containerName="payload-processing"
Nov 03 18:09:26.781474 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:26.781465    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="05885de3-0afb-4ae7-a7d2-7c3b64e0bba1" containerName="payload-pre-processing"
Nov 03 18:09:26.781545 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:26.781536    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dd7528b-e815-44b0-aa17-72f16301468e" containerName="payload-pre-processing"
Nov 03 18:09:26.781643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:26.781631    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ced2f03-d1ad-4a25-aa71-d6ece3178be9" containerName="payload-processing"
Nov 03 18:09:26.786937 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:26.786911    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c91bee-76b997676c-zfjvq"
Nov 03 18:09:26.790931 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:26.790908    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-c91bee-76b997676c-zfjvq"]
Nov 03 18:09:26.795817 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:26.795794    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4b6e7259-854d546fff-j5qbr"]
Nov 03 18:09:26.801105 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:26.801030    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4b6e7259-854d546fff-j5qbr"]
Nov 03 18:09:26.806135 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:26.806115    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c91bee-687764d75f-5dlqr"]
Nov 03 18:09:26.806935 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:26.806919    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="8832b3d8-1880-4277-8a0c-f9bf94eef7b3" containerName="istio-proxy"
Nov 03 18:09:26.809444 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:26.809427    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c91bee-687764d75f-5dlqr"
Nov 03 18:09:26.811036 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:26.811013    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c91bee-687764d75f-5dlqr"]
Nov 03 18:09:26.836021 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:26.835998    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-c91bee\""
Nov 03 18:09:26.836230 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:26.836216    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-c91bee-dockercfg-rbznp\""
Nov 03 18:09:26.903775 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:26.903735    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dgn5\" (UniqueName: \"kubernetes.io/projected/1978e992-0367-40cf-a9f0-7899ab8215c9-kube-api-access-5dgn5\") pod \"payload-processing-e2e-shared-a-w4-c91bee-76b997676c-zfjvq\" (UID: \"1978e992-0367-40cf-a9f0-7899ab8215c9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c91bee-76b997676c-zfjvq"
Nov 03 18:09:26.904041 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:26.904018    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16103631-17f7-429b-b648-49691db9e8a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-c91bee-687764d75f-5dlqr\" (UID: \"16103631-17f7-429b-b648-49691db9e8a5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c91bee-687764d75f-5dlqr"
Nov 03 18:09:26.904164 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:26.904150    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1978e992-0367-40cf-a9f0-7899ab8215c9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-c91bee-76b997676c-zfjvq\" (UID: \"1978e992-0367-40cf-a9f0-7899ab8215c9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c91bee-76b997676c-zfjvq"
Nov 03 18:09:26.904243 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:26.904216    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vshlm\" (UniqueName: \"kubernetes.io/projected/16103631-17f7-429b-b648-49691db9e8a5-kube-api-access-vshlm\") pod \"payload-pre-processing-e2e-shared-a-w4-c91bee-687764d75f-5dlqr\" (UID: \"16103631-17f7-429b-b648-49691db9e8a5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c91bee-687764d75f-5dlqr"
Nov 03 18:09:27.005634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:27.005524    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1978e992-0367-40cf-a9f0-7899ab8215c9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-c91bee-76b997676c-zfjvq\" (UID: \"1978e992-0367-40cf-a9f0-7899ab8215c9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c91bee-76b997676c-zfjvq"
Nov 03 18:09:27.005817 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:27.005672    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vshlm\" (UniqueName: \"kubernetes.io/projected/16103631-17f7-429b-b648-49691db9e8a5-kube-api-access-vshlm\") pod \"payload-pre-processing-e2e-shared-a-w4-c91bee-687764d75f-5dlqr\" (UID: \"16103631-17f7-429b-b648-49691db9e8a5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c91bee-687764d75f-5dlqr"
Nov 03 18:09:27.005817 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:27.005722    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dgn5\" (UniqueName: \"kubernetes.io/projected/1978e992-0367-40cf-a9f0-7899ab8215c9-kube-api-access-5dgn5\") pod \"payload-processing-e2e-shared-a-w4-c91bee-76b997676c-zfjvq\" (UID: \"1978e992-0367-40cf-a9f0-7899ab8215c9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c91bee-76b997676c-zfjvq"
Nov 03 18:09:27.005989 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:27.005961    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16103631-17f7-429b-b648-49691db9e8a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-c91bee-687764d75f-5dlqr\" (UID: \"16103631-17f7-429b-b648-49691db9e8a5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c91bee-687764d75f-5dlqr"
Nov 03 18:09:27.006331 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:27.006310    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1978e992-0367-40cf-a9f0-7899ab8215c9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-c91bee-76b997676c-zfjvq\" (UID: \"1978e992-0367-40cf-a9f0-7899ab8215c9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c91bee-76b997676c-zfjvq"
Nov 03 18:09:27.006683 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:27.006665    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16103631-17f7-429b-b648-49691db9e8a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-c91bee-687764d75f-5dlqr\" (UID: \"16103631-17f7-429b-b648-49691db9e8a5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c91bee-687764d75f-5dlqr"
Nov 03 18:09:27.063184 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:27.063153    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vshlm\" (UniqueName: \"kubernetes.io/projected/16103631-17f7-429b-b648-49691db9e8a5-kube-api-access-vshlm\") pod \"payload-pre-processing-e2e-shared-a-w4-c91bee-687764d75f-5dlqr\" (UID: \"16103631-17f7-429b-b648-49691db9e8a5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c91bee-687764d75f-5dlqr"
Nov 03 18:09:27.063540 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:27.063381    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dgn5\" (UniqueName: \"kubernetes.io/projected/1978e992-0367-40cf-a9f0-7899ab8215c9-kube-api-access-5dgn5\") pod \"payload-processing-e2e-shared-a-w4-c91bee-76b997676c-zfjvq\" (UID: \"1978e992-0367-40cf-a9f0-7899ab8215c9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c91bee-76b997676c-zfjvq"
Nov 03 18:09:27.102870 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:27.102840    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c91bee-76b997676c-zfjvq"
Nov 03 18:09:27.106095 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:27.106070    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dd7528b-e815-44b0-aa17-72f16301468e" path="/var/lib/kubelet/pods/1dd7528b-e815-44b0-aa17-72f16301468e/volumes"
Nov 03 18:09:27.106584 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:27.106568    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdafb581-457b-4a16-a9b7-9c19f90f30cd" path="/var/lib/kubelet/pods/fdafb581-457b-4a16-a9b7-9c19f90f30cd/volumes"
Nov 03 18:09:27.121716 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:27.121688    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c91bee-687764d75f-5dlqr"
Nov 03 18:09:27.497503 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:09:27.497376    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1978e992_0367_40cf_a9f0_7899ab8215c9.slice/crio-64bd3c2af8341fb1b5a2943fe207f0366f6618cdc561476ec687674845561141 WatchSource:0}: Error finding container 64bd3c2af8341fb1b5a2943fe207f0366f6618cdc561476ec687674845561141: Status 404 returned error can't find the container with id 64bd3c2af8341fb1b5a2943fe207f0366f6618cdc561476ec687674845561141
Nov 03 18:09:27.538375 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:27.538277    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-c91bee-76b997676c-zfjvq"]
Nov 03 18:09:27.552185 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:09:27.552132    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16103631_17f7_429b_b648_49691db9e8a5.slice/crio-627c76fba9566cee340847f1d33c3e1e06df86f1cc3a49c3a194e3ad8a2933f4 WatchSource:0}: Error finding container 627c76fba9566cee340847f1d33c3e1e06df86f1cc3a49c3a194e3ad8a2933f4: Status 404 returned error can't find the container with id 627c76fba9566cee340847f1d33c3e1e06df86f1cc3a49c3a194e3ad8a2933f4
Nov 03 18:09:27.609156 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:27.607768    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c91bee-687764d75f-5dlqr"]
Nov 03 18:09:28.035717 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.035669    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c91bee-687764d75f-5dlqr" event={"ID":"16103631-17f7-429b-b648-49691db9e8a5","Type":"ContainerStarted","Data":"2ca248c204797b3742e615508f42cced098bba4bbf8b2bc9393a702e545617dc"}
Nov 03 18:09:28.035883 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.035740    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c91bee-687764d75f-5dlqr" event={"ID":"16103631-17f7-429b-b648-49691db9e8a5","Type":"ContainerStarted","Data":"627c76fba9566cee340847f1d33c3e1e06df86f1cc3a49c3a194e3ad8a2933f4"}
Nov 03 18:09:28.035883 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.035770    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c91bee-687764d75f-5dlqr"
Nov 03 18:09:28.037474 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.037445    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c91bee-76b997676c-zfjvq" event={"ID":"1978e992-0367-40cf-a9f0-7899ab8215c9","Type":"ContainerStarted","Data":"7297a1e5f173f8e7c0c8aa3aa585a358bd4e3c94132dc99dcb551298c51a49ba"}
Nov 03 18:09:28.037570 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.037506    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c91bee-76b997676c-zfjvq"
Nov 03 18:09:28.037570 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.037533    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c91bee-76b997676c-zfjvq" event={"ID":"1978e992-0367-40cf-a9f0-7899ab8215c9","Type":"ContainerStarted","Data":"64bd3c2af8341fb1b5a2943fe207f0366f6618cdc561476ec687674845561141"}
Nov 03 18:09:28.147076 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.147020    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c91bee-687764d75f-5dlqr" podStartSLOduration=2.147006054 podStartE2EDuration="2.147006054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:09:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:09:28.143594737 +0000 UTC m=+1407.159792806" watchObservedRunningTime="2026-11-03 18:09:28.147006054 +0000 UTC m=+1407.163204123"
Nov 03 18:09:28.257071 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.257010    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c91bee-76b997676c-zfjvq" podStartSLOduration=2.25692184 podStartE2EDuration="2.25692184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:09:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:09:28.251399786 +0000 UTC m=+1407.267597855" watchObservedRunningTime="2026-11-03 18:09:28.25692184 +0000 UTC m=+1407.273119910"
Nov 03 18:09:28.904818 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.904786    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt"
Nov 03 18:09:28.911281 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.911255    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg"
Nov 03 18:09:28.944617 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.944578    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-istio-envoy\") pod \"86e80b99-3e6f-41ae-8832-9cfbbf2e543a\" (UID: \"86e80b99-3e6f-41ae-8832-9cfbbf2e543a\") "
Nov 03 18:09:28.944752 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.944694    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feb037bd-264e-486b-aa7d-f3679b439981-kube-api-access-hww2q\" (UniqueName: \"kubernetes.io/projected/feb037bd-264e-486b-aa7d-f3679b439981-kube-api-access-hww2q\") pod \"feb037bd-264e-486b-aa7d-f3679b439981\" (UID: \"feb037bd-264e-486b-aa7d-f3679b439981\") "
Nov 03 18:09:28.944752 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.944725    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feb037bd-264e-486b-aa7d-f3679b439981-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/feb037bd-264e-486b-aa7d-f3679b439981-istio-envoy\") pod \"feb037bd-264e-486b-aa7d-f3679b439981\" (UID: \"feb037bd-264e-486b-aa7d-f3679b439981\") "
Nov 03 18:09:28.944752 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.944744    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/feb037bd-264e-486b-aa7d-f3679b439981-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/feb037bd-264e-486b-aa7d-f3679b439981-istiod-ca-cert\") pod \"feb037bd-264e-486b-aa7d-f3679b439981\" (UID: \"feb037bd-264e-486b-aa7d-f3679b439981\") "
Nov 03 18:09:28.944880 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.944767    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-istio-data\") pod \"86e80b99-3e6f-41ae-8832-9cfbbf2e543a\" (UID: \"86e80b99-3e6f-41ae-8832-9cfbbf2e543a\") "
Nov 03 18:09:28.944880 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.944837    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feb037bd-264e-486b-aa7d-f3679b439981-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/feb037bd-264e-486b-aa7d-f3679b439981-istio-data\") pod \"feb037bd-264e-486b-aa7d-f3679b439981\" (UID: \"feb037bd-264e-486b-aa7d-f3679b439981\") "
Nov 03 18:09:28.944986 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.944887    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feb037bd-264e-486b-aa7d-f3679b439981-istio-token\" (UniqueName: \"kubernetes.io/projected/feb037bd-264e-486b-aa7d-f3679b439981-istio-token\") pod \"feb037bd-264e-486b-aa7d-f3679b439981\" (UID: \"feb037bd-264e-486b-aa7d-f3679b439981\") "
Nov 03 18:09:28.944986 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.944952    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-credential-socket\") pod \"86e80b99-3e6f-41ae-8832-9cfbbf2e543a\" (UID: \"86e80b99-3e6f-41ae-8832-9cfbbf2e543a\") "
Nov 03 18:09:28.945130 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.945093    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-kube-api-access-96wb5\" (UniqueName: \"kubernetes.io/projected/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-kube-api-access-96wb5\") pod \"86e80b99-3e6f-41ae-8832-9cfbbf2e543a\" (UID: \"86e80b99-3e6f-41ae-8832-9cfbbf2e543a\") "
Nov 03 18:09:28.945130 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.945110    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-istio-data" pod "86e80b99-3e6f-41ae-8832-9cfbbf2e543a" (UID: "86e80b99-3e6f-41ae-8832-9cfbbf2e543a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:28.945246 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.945166    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-workload-certs\") pod \"86e80b99-3e6f-41ae-8832-9cfbbf2e543a\" (UID: \"86e80b99-3e6f-41ae-8832-9cfbbf2e543a\") "
Nov 03 18:09:28.945246 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.945161    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feb037bd-264e-486b-aa7d-f3679b439981-istiod-ca-cert" pod "feb037bd-264e-486b-aa7d-f3679b439981" (UID: "feb037bd-264e-486b-aa7d-f3679b439981"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:09:28.945246 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.945240    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-istiod-ca-cert\") pod \"86e80b99-3e6f-41ae-8832-9cfbbf2e543a\" (UID: \"86e80b99-3e6f-41ae-8832-9cfbbf2e543a\") "
Nov 03 18:09:28.945400 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.945280    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-istio-podinfo\") pod \"86e80b99-3e6f-41ae-8832-9cfbbf2e543a\" (UID: \"86e80b99-3e6f-41ae-8832-9cfbbf2e543a\") "
Nov 03 18:09:28.945400 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.945323    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feb037bd-264e-486b-aa7d-f3679b439981-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/feb037bd-264e-486b-aa7d-f3679b439981-workload-socket\") pod \"feb037bd-264e-486b-aa7d-f3679b439981\" (UID: \"feb037bd-264e-486b-aa7d-f3679b439981\") "
Nov 03 18:09:28.945509 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.945405    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/feb037bd-264e-486b-aa7d-f3679b439981-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/feb037bd-264e-486b-aa7d-f3679b439981-istio-podinfo\") pod \"feb037bd-264e-486b-aa7d-f3679b439981\" (UID: \"feb037bd-264e-486b-aa7d-f3679b439981\") "
Nov 03 18:09:28.945509 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.945421    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feb037bd-264e-486b-aa7d-f3679b439981-istio-data" pod "feb037bd-264e-486b-aa7d-f3679b439981" (UID: "feb037bd-264e-486b-aa7d-f3679b439981"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:28.945509 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.945460    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-istio-token\" (UniqueName: \"kubernetes.io/projected/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-istio-token\") pod \"86e80b99-3e6f-41ae-8832-9cfbbf2e543a\" (UID: \"86e80b99-3e6f-41ae-8832-9cfbbf2e543a\") "
Nov 03 18:09:28.945509 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.945480    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-credential-socket" pod "86e80b99-3e6f-41ae-8832-9cfbbf2e543a" (UID: "86e80b99-3e6f-41ae-8832-9cfbbf2e543a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:28.945740 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.945523    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feb037bd-264e-486b-aa7d-f3679b439981-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/feb037bd-264e-486b-aa7d-f3679b439981-credential-socket\") pod \"feb037bd-264e-486b-aa7d-f3679b439981\" (UID: \"feb037bd-264e-486b-aa7d-f3679b439981\") "
Nov 03 18:09:28.945740 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.945558    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-workload-socket\") pod \"86e80b99-3e6f-41ae-8832-9cfbbf2e543a\" (UID: \"86e80b99-3e6f-41ae-8832-9cfbbf2e543a\") "
Nov 03 18:09:28.945740 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.945599    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feb037bd-264e-486b-aa7d-f3679b439981-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/feb037bd-264e-486b-aa7d-f3679b439981-workload-certs\") pod \"feb037bd-264e-486b-aa7d-f3679b439981\" (UID: \"feb037bd-264e-486b-aa7d-f3679b439981\") "
Nov 03 18:09:28.945740 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.945690    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feb037bd-264e-486b-aa7d-f3679b439981-workload-socket" pod "feb037bd-264e-486b-aa7d-f3679b439981" (UID: "feb037bd-264e-486b-aa7d-f3679b439981"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:28.945946 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.945760    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-istiod-ca-cert" pod "86e80b99-3e6f-41ae-8832-9cfbbf2e543a" (UID: "86e80b99-3e6f-41ae-8832-9cfbbf2e543a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:09:28.946053 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.946030    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-workload-certs" pod "86e80b99-3e6f-41ae-8832-9cfbbf2e543a" (UID: "86e80b99-3e6f-41ae-8832-9cfbbf2e543a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:28.946758 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.946580    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feb037bd-264e-486b-aa7d-f3679b439981-credential-socket" pod "feb037bd-264e-486b-aa7d-f3679b439981" (UID: "feb037bd-264e-486b-aa7d-f3679b439981"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:28.946758 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.946652    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/feb037bd-264e-486b-aa7d-f3679b439981-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:28.946758 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.946680    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:28.946758 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.946697    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/feb037bd-264e-486b-aa7d-f3679b439981-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:28.946758 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.946712    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:28.946758 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.946726    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:28.946758 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.946739    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:28.946758 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.946753    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/feb037bd-264e-486b-aa7d-f3679b439981-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:28.947236 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.946926    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-workload-socket" pod "86e80b99-3e6f-41ae-8832-9cfbbf2e543a" (UID: "86e80b99-3e6f-41ae-8832-9cfbbf2e543a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:28.947236 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.947029    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feb037bd-264e-486b-aa7d-f3679b439981-workload-certs" pod "feb037bd-264e-486b-aa7d-f3679b439981" (UID: "feb037bd-264e-486b-aa7d-f3679b439981"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:28.947512 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.947485    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feb037bd-264e-486b-aa7d-f3679b439981-kube-api-access-hww2q" pod "feb037bd-264e-486b-aa7d-f3679b439981" (UID: "feb037bd-264e-486b-aa7d-f3679b439981"). InnerVolumeSpecName "kube-api-access-hww2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:09:28.947932 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.947907    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feb037bd-264e-486b-aa7d-f3679b439981-istio-envoy" pod "feb037bd-264e-486b-aa7d-f3679b439981" (UID: "feb037bd-264e-486b-aa7d-f3679b439981"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:28.948061 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.948036    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-istio-podinfo" pod "86e80b99-3e6f-41ae-8832-9cfbbf2e543a" (UID: "86e80b99-3e6f-41ae-8832-9cfbbf2e543a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:09:28.948149 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.948057    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-kube-api-access-96wb5" pod "86e80b99-3e6f-41ae-8832-9cfbbf2e543a" (UID: "86e80b99-3e6f-41ae-8832-9cfbbf2e543a"). InnerVolumeSpecName "kube-api-access-96wb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:09:28.948399 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.948381    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-istio-envoy" pod "86e80b99-3e6f-41ae-8832-9cfbbf2e543a" (UID: "86e80b99-3e6f-41ae-8832-9cfbbf2e543a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:28.948755 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.948738    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feb037bd-264e-486b-aa7d-f3679b439981-istio-token" pod "feb037bd-264e-486b-aa7d-f3679b439981" (UID: "feb037bd-264e-486b-aa7d-f3679b439981"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:09:28.948845 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.948830    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/feb037bd-264e-486b-aa7d-f3679b439981-istio-podinfo" pod "feb037bd-264e-486b-aa7d-f3679b439981" (UID: "feb037bd-264e-486b-aa7d-f3679b439981"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:09:28.949276 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:28.949263    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-istio-token" pod "86e80b99-3e6f-41ae-8832-9cfbbf2e543a" (UID: "86e80b99-3e6f-41ae-8832-9cfbbf2e543a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:09:29.048658 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:29.048605    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:29.048658 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:29.048660    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/feb037bd-264e-486b-aa7d-f3679b439981-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:29.048875 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:29.048675    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:29.048875 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:29.048690    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/feb037bd-264e-486b-aa7d-f3679b439981-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:29.048875 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:29.048704    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:29.048875 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:29.048718    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/feb037bd-264e-486b-aa7d-f3679b439981-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:29.048875 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:29.048731    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:29.048875 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:29.048745    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hww2q\" (UniqueName: \"kubernetes.io/projected/feb037bd-264e-486b-aa7d-f3679b439981-kube-api-access-hww2q\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:29.048875 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:29.048759    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/feb037bd-264e-486b-aa7d-f3679b439981-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:29.048875 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:29.048774    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/feb037bd-264e-486b-aa7d-f3679b439981-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:29.048875 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:29.048787    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96wb5\" (UniqueName: \"kubernetes.io/projected/86e80b99-3e6f-41ae-8832-9cfbbf2e543a-kube-api-access-96wb5\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:29.050152 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:29.050123    5315 generic.go:356] "Generic (PLEG): container finished" podID="feb037bd-264e-486b-aa7d-f3679b439981" containerID="51097ba4de28cb7a57daccd00acaff578898ba5cf1f9164d51ed3177999e2ed7" exitCode=0
Nov 03 18:09:29.050290 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:29.050201    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt"
Nov 03 18:09:29.050290 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:29.050214    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt" event={"ID":"feb037bd-264e-486b-aa7d-f3679b439981","Type":"ContainerDied","Data":"51097ba4de28cb7a57daccd00acaff578898ba5cf1f9164d51ed3177999e2ed7"}
Nov 03 18:09:29.050417 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:29.050317    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt" event={"ID":"feb037bd-264e-486b-aa7d-f3679b439981","Type":"ContainerDied","Data":"53dcaba89d5a00dd436e66a353d87e176b19c1d0e23afcfceec680e7f6e0ff7d"}
Nov 03 18:09:29.050417 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:29.050337    5315 scope.go:122] "RemoveContainer" containerID="51097ba4de28cb7a57daccd00acaff578898ba5cf1f9164d51ed3177999e2ed7"
Nov 03 18:09:29.052105 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:29.052082    5315 generic.go:356] "Generic (PLEG): container finished" podID="86e80b99-3e6f-41ae-8832-9cfbbf2e543a" containerID="0dfdd1466f5ab8b88a3e96c4d98823ffdbe87503efd084a453c49b0d9a80e2cf" exitCode=0
Nov 03 18:09:29.052908 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:29.052888    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg"
Nov 03 18:09:29.054163 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:29.054135    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg" event={"ID":"86e80b99-3e6f-41ae-8832-9cfbbf2e543a","Type":"ContainerDied","Data":"0dfdd1466f5ab8b88a3e96c4d98823ffdbe87503efd084a453c49b0d9a80e2cf"}
Nov 03 18:09:29.054272 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:29.054189    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg" event={"ID":"86e80b99-3e6f-41ae-8832-9cfbbf2e543a","Type":"ContainerDied","Data":"654b5fdf6fe85e3ddf58b6c3f2eaf7b2b7a6fa50235006ca55c7af9abc834097"}
Nov 03 18:09:29.066071 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:29.066052    5315 scope.go:122] "RemoveContainer" containerID="51097ba4de28cb7a57daccd00acaff578898ba5cf1f9164d51ed3177999e2ed7"
Nov 03 18:09:29.066418 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:09:29.066394    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51097ba4de28cb7a57daccd00acaff578898ba5cf1f9164d51ed3177999e2ed7\": container with ID starting with 51097ba4de28cb7a57daccd00acaff578898ba5cf1f9164d51ed3177999e2ed7 not found: ID does not exist" containerID="51097ba4de28cb7a57daccd00acaff578898ba5cf1f9164d51ed3177999e2ed7"
Nov 03 18:09:29.066517 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:29.066428    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51097ba4de28cb7a57daccd00acaff578898ba5cf1f9164d51ed3177999e2ed7"} err="failed to get container status \"51097ba4de28cb7a57daccd00acaff578898ba5cf1f9164d51ed3177999e2ed7\": rpc error: code = NotFound desc = could not find container \"51097ba4de28cb7a57daccd00acaff578898ba5cf1f9164d51ed3177999e2ed7\": container with ID starting with 51097ba4de28cb7a57daccd00acaff578898ba5cf1f9164d51ed3177999e2ed7 not found: ID does not exist"
Nov 03 18:09:29.066517 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:29.066452    5315 scope.go:122] "RemoveContainer" containerID="0dfdd1466f5ab8b88a3e96c4d98823ffdbe87503efd084a453c49b0d9a80e2cf"
Nov 03 18:09:29.114673 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:29.111025    5315 scope.go:122] "RemoveContainer" containerID="0dfdd1466f5ab8b88a3e96c4d98823ffdbe87503efd084a453c49b0d9a80e2cf"
Nov 03 18:09:29.114673 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:09:29.112086    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dfdd1466f5ab8b88a3e96c4d98823ffdbe87503efd084a453c49b0d9a80e2cf\": container with ID starting with 0dfdd1466f5ab8b88a3e96c4d98823ffdbe87503efd084a453c49b0d9a80e2cf not found: ID does not exist" containerID="0dfdd1466f5ab8b88a3e96c4d98823ffdbe87503efd084a453c49b0d9a80e2cf"
Nov 03 18:09:29.114673 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:29.112129    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dfdd1466f5ab8b88a3e96c4d98823ffdbe87503efd084a453c49b0d9a80e2cf"} err="failed to get container status \"0dfdd1466f5ab8b88a3e96c4d98823ffdbe87503efd084a453c49b0d9a80e2cf\": rpc error: code = NotFound desc = could not find container \"0dfdd1466f5ab8b88a3e96c4d98823ffdbe87503efd084a453c49b0d9a80e2cf\": container with ID starting with 0dfdd1466f5ab8b88a3e96c4d98823ffdbe87503efd084a453c49b0d9a80e2cf not found: ID does not exist"
Nov 03 18:09:29.336992 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:29.336948    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg"]
Nov 03 18:09:29.342710 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:29.342685    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-lglgg"]
Nov 03 18:09:29.445011 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:29.444979    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt"]
Nov 03 18:09:29.450371 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:29.450339    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-7zmvt"]
Nov 03 18:09:31.279056 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:31.279018    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86e80b99-3e6f-41ae-8832-9cfbbf2e543a" path="/var/lib/kubelet/pods/86e80b99-3e6f-41ae-8832-9cfbbf2e543a/volumes"
Nov 03 18:09:31.279880 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:31.279857    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="feb037bd-264e-486b-aa7d-f3679b439981" path="/var/lib/kubelet/pods/feb037bd-264e-486b-aa7d-f3679b439981/volumes"
Nov 03 18:09:31.403984 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:31.403947    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5ffd208e-5c84c4495f-nk9hq"]
Nov 03 18:09:31.404697 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:31.404679    5315 cpu_manager.go:414] "containerMap: removing container" podUID="feb037bd-264e-486b-aa7d-f3679b439981" containerName="istio-proxy"
Nov 03 18:09:31.404697 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:31.404698    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feb037bd-264e-486b-aa7d-f3679b439981" containerName="istio-proxy"
Nov 03 18:09:31.404832 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:31.404719    5315 cpu_manager.go:414] "containerMap: removing container" podUID="86e80b99-3e6f-41ae-8832-9cfbbf2e543a" containerName="istio-proxy"
Nov 03 18:09:31.404832 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:31.404725    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86e80b99-3e6f-41ae-8832-9cfbbf2e543a" containerName="istio-proxy"
Nov 03 18:09:31.404920 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:31.404889    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="feb037bd-264e-486b-aa7d-f3679b439981" containerName="istio-proxy"
Nov 03 18:09:31.404920 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:31.404901    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="86e80b99-3e6f-41ae-8832-9cfbbf2e543a" containerName="istio-proxy"
Nov 03 18:09:31.427633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:31.424167    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-5ffd208e-5c84c4495f-nk9hq"
Nov 03 18:09:31.519633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:31.519562    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-5ffd208e\""
Nov 03 18:09:31.535685 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:31.519860    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-5ffd208e-dockercfg-ck8s6\""
Nov 03 18:09:31.535685 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:31.522948    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9b0b852-1183-4ae0-87a0-6832aa16c3d9-plugins-config-volume\") pod \"payload-processing-e2e-derive-5ffd208e-5c84c4495f-nk9hq\" (UID: \"d9b0b852-1183-4ae0-87a0-6832aa16c3d9\") " pod="openshift-ingress/payload-processing-e2e-derive-5ffd208e-5c84c4495f-nk9hq"
Nov 03 18:09:31.535685 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:31.523259    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmbtr\" (UniqueName: \"kubernetes.io/projected/d9b0b852-1183-4ae0-87a0-6832aa16c3d9-kube-api-access-lmbtr\") pod \"payload-processing-e2e-derive-5ffd208e-5c84c4495f-nk9hq\" (UID: \"d9b0b852-1183-4ae0-87a0-6832aa16c3d9\") " pod="openshift-ingress/payload-processing-e2e-derive-5ffd208e-5c84c4495f-nk9hq"
Nov 03 18:09:31.838686 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:31.837752    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5ffd208e-5c84c4495f-nk9hq"]
Nov 03 18:09:31.894096 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:31.893436    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9b0b852-1183-4ae0-87a0-6832aa16c3d9-plugins-config-volume\") pod \"payload-processing-e2e-derive-5ffd208e-5c84c4495f-nk9hq\" (UID: \"d9b0b852-1183-4ae0-87a0-6832aa16c3d9\") " pod="openshift-ingress/payload-processing-e2e-derive-5ffd208e-5c84c4495f-nk9hq"
Nov 03 18:09:31.894096 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:31.893874    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmbtr\" (UniqueName: \"kubernetes.io/projected/d9b0b852-1183-4ae0-87a0-6832aa16c3d9-kube-api-access-lmbtr\") pod \"payload-processing-e2e-derive-5ffd208e-5c84c4495f-nk9hq\" (UID: \"d9b0b852-1183-4ae0-87a0-6832aa16c3d9\") " pod="openshift-ingress/payload-processing-e2e-derive-5ffd208e-5c84c4495f-nk9hq"
Nov 03 18:09:31.895337 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:31.895311    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9b0b852-1183-4ae0-87a0-6832aa16c3d9-plugins-config-volume\") pod \"payload-processing-e2e-derive-5ffd208e-5c84c4495f-nk9hq\" (UID: \"d9b0b852-1183-4ae0-87a0-6832aa16c3d9\") " pod="openshift-ingress/payload-processing-e2e-derive-5ffd208e-5c84c4495f-nk9hq"
Nov 03 18:09:31.903809 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:31.903781    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5ffd208e-64b984695-8jftk"]
Nov 03 18:09:31.911708 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:31.911680    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5ffd208e-64b984695-8jftk"
Nov 03 18:09:31.914336 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:31.914299    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5ffd208e-64b984695-8jftk"]
Nov 03 18:09:31.951194 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:31.951171    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmbtr\" (UniqueName: \"kubernetes.io/projected/d9b0b852-1183-4ae0-87a0-6832aa16c3d9-kube-api-access-lmbtr\") pod \"payload-processing-e2e-derive-5ffd208e-5c84c4495f-nk9hq\" (UID: \"d9b0b852-1183-4ae0-87a0-6832aa16c3d9\") " pod="openshift-ingress/payload-processing-e2e-derive-5ffd208e-5c84c4495f-nk9hq"
Nov 03 18:09:31.995745 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:31.995705    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79925de9-7aba-4361-bc74-40ecc8f6bee2-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-5ffd208e-64b984695-8jftk\" (UID: \"79925de9-7aba-4361-bc74-40ecc8f6bee2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5ffd208e-64b984695-8jftk"
Nov 03 18:09:31.995904 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:31.995844    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh2rc\" (UniqueName: \"kubernetes.io/projected/79925de9-7aba-4361-bc74-40ecc8f6bee2-kube-api-access-wh2rc\") pod \"payload-pre-processing-e2e-derive-5ffd208e-64b984695-8jftk\" (UID: \"79925de9-7aba-4361-bc74-40ecc8f6bee2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5ffd208e-64b984695-8jftk"
Nov 03 18:09:32.082823 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:32.082786    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-5ffd208e-5c84c4495f-nk9hq"
Nov 03 18:09:32.106161 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:32.100817    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79925de9-7aba-4361-bc74-40ecc8f6bee2-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-5ffd208e-64b984695-8jftk\" (UID: \"79925de9-7aba-4361-bc74-40ecc8f6bee2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5ffd208e-64b984695-8jftk"
Nov 03 18:09:32.106161 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:32.100928    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh2rc\" (UniqueName: \"kubernetes.io/projected/79925de9-7aba-4361-bc74-40ecc8f6bee2-kube-api-access-wh2rc\") pod \"payload-pre-processing-e2e-derive-5ffd208e-64b984695-8jftk\" (UID: \"79925de9-7aba-4361-bc74-40ecc8f6bee2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5ffd208e-64b984695-8jftk"
Nov 03 18:09:32.106161 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:32.103663    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79925de9-7aba-4361-bc74-40ecc8f6bee2-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-5ffd208e-64b984695-8jftk\" (UID: \"79925de9-7aba-4361-bc74-40ecc8f6bee2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5ffd208e-64b984695-8jftk"
Nov 03 18:09:32.159408 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:32.159365    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh2rc\" (UniqueName: \"kubernetes.io/projected/79925de9-7aba-4361-bc74-40ecc8f6bee2-kube-api-access-wh2rc\") pod \"payload-pre-processing-e2e-derive-5ffd208e-64b984695-8jftk\" (UID: \"79925de9-7aba-4361-bc74-40ecc8f6bee2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5ffd208e-64b984695-8jftk"
Nov 03 18:09:32.231140 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:32.231099    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5ffd208e-64b984695-8jftk"
Nov 03 18:09:32.539429 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:32.539243    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5ffd208e-5c84c4495f-nk9hq"]
Nov 03 18:09:32.542817 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:09:32.542783    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9b0b852_1183_4ae0_87a0_6832aa16c3d9.slice/crio-2f8fa6e0daafc3e04ecd37970a19662bebaa3a3e1879280e7bc2932cc0abfe7d WatchSource:0}: Error finding container 2f8fa6e0daafc3e04ecd37970a19662bebaa3a3e1879280e7bc2932cc0abfe7d: Status 404 returned error can't find the container with id 2f8fa6e0daafc3e04ecd37970a19662bebaa3a3e1879280e7bc2932cc0abfe7d
Nov 03 18:09:32.652629 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:32.652589    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5ffd208e-64b984695-8jftk"]
Nov 03 18:09:32.656140 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:09:32.656103    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79925de9_7aba_4361_bc74_40ecc8f6bee2.slice/crio-28f6547803261d1bd8c3932755eb801c0836a3b7b2f681643ca7236e2bc796f8 WatchSource:0}: Error finding container 28f6547803261d1bd8c3932755eb801c0836a3b7b2f681643ca7236e2bc796f8: Status 404 returned error can't find the container with id 28f6547803261d1bd8c3932755eb801c0836a3b7b2f681643ca7236e2bc796f8
Nov 03 18:09:33.189395 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:33.189348    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-5ffd208e-5c84c4495f-nk9hq" event={"ID":"d9b0b852-1183-4ae0-87a0-6832aa16c3d9","Type":"ContainerStarted","Data":"6591c34c4d66ebcd678cbeca98781d21919cd7fdc2c38981ca3b11255a913f94"}
Nov 03 18:09:33.189630 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:33.189426    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-5ffd208e-5c84c4495f-nk9hq" event={"ID":"d9b0b852-1183-4ae0-87a0-6832aa16c3d9","Type":"ContainerStarted","Data":"2f8fa6e0daafc3e04ecd37970a19662bebaa3a3e1879280e7bc2932cc0abfe7d"}
Nov 03 18:09:33.189630 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:33.189460    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-5ffd208e-5c84c4495f-nk9hq"
Nov 03 18:09:33.191059 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:33.191035    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5ffd208e-64b984695-8jftk" event={"ID":"79925de9-7aba-4361-bc74-40ecc8f6bee2","Type":"ContainerStarted","Data":"4ae8d7f08e897610285ec25ee70df1b8580205cb0ab3dbb97d8d015da666e0a9"}
Nov 03 18:09:33.191172 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:33.191078    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5ffd208e-64b984695-8jftk" event={"ID":"79925de9-7aba-4361-bc74-40ecc8f6bee2","Type":"ContainerStarted","Data":"28f6547803261d1bd8c3932755eb801c0836a3b7b2f681643ca7236e2bc796f8"}
Nov 03 18:09:33.191172 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:33.191099    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-5ffd208e-64b984695-8jftk"
Nov 03 18:09:33.298835 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:33.298777    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-5ffd208e-5c84c4495f-nk9hq" podStartSLOduration=2.298762323 podStartE2EDuration="2.298762323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:09:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:09:33.296587029 +0000 UTC m=+1412.312785100" watchObservedRunningTime="2026-11-03 18:09:33.298762323 +0000 UTC m=+1412.314960391"
Nov 03 18:09:33.404397 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:33.404340    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-5ffd208e-64b984695-8jftk" podStartSLOduration=2.4043270310000002 podStartE2EDuration="2.404327031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:09:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:09:33.403890919 +0000 UTC m=+1412.420088988" watchObservedRunningTime="2026-11-03 18:09:33.404327031 +0000 UTC m=+1412.420525100"
Nov 03 18:09:39.056299 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:39.056263    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c91bee-687764d75f-5dlqr"
Nov 03 18:09:39.056835 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:39.056385    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c91bee-76b997676c-zfjvq"
Nov 03 18:09:42.433413 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:42.433377    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-248f8ee3-rvrks"]
Nov 03 18:09:42.439622 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:42.439582    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-248f8ee3-rvrks"]
Nov 03 18:09:43.106398 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:43.106362    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="522d42d2-228a-4242-bd5f-b2e3074b7b89" path="/var/lib/kubelet/pods/522d42d2-228a-4242-bd5f-b2e3074b7b89/volumes"
Nov 03 18:09:44.034483 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:44.034431    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q"]
Nov 03 18:09:44.035027 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:44.034750    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q" podUID="11671eb9-4824-4e89-9692-b5be9038c549" containerName="istio-proxy" containerID="cri-o://12f02d737977cbccd0d09199b884317ef35871693aca2a93336aefa5081dfdb3" gracePeriod=30
Nov 03 18:09:44.260678 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:44.255813    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-5ffd208e-5c84c4495f-nk9hq"
Nov 03 18:09:44.260678 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:44.257277    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-5ffd208e-64b984695-8jftk"
Nov 03 18:09:45.788673 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:45.788603    5315 patch_prober.go:29] interesting pod/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:09:45.789075 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:45.788696    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q" podUID="11671eb9-4824-4e89-9692-b5be9038c549" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:09:48.929722 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:48.929543    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b6e7259-fwzg9"]
Nov 03 18:09:48.990675 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:48.990636    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b6e7259-fwzg9"]
Nov 03 18:09:49.091957 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:49.091914    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b01870c-v92xt"]
Nov 03 18:09:49.098896 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:49.098867    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b01870c-v92xt"
Nov 03 18:09:49.110172 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:49.110143    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="885b7b64-07d4-4d24-b961-77841e1f4ba8" path="/var/lib/kubelet/pods/885b7b64-07d4-4d24-b961-77841e1f4ba8/volumes"
Nov 03 18:09:49.150205 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:49.150084    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:09:49.150205 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:49.150165    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:09:49.233029 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:49.203903    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:09:49.233029 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:49.204058    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:09:49.233029 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:49.226737    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81f2c5ce-15fa-4010-b323-5cddd39d5667-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b01870c-v92xt\" (UID: \"81f2c5ce-15fa-4010-b323-5cddd39d5667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b01870c-v92xt"
Nov 03 18:09:49.233029 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:49.226952    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81f2c5ce-15fa-4010-b323-5cddd39d5667-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b01870c-v92xt\" (UID: \"81f2c5ce-15fa-4010-b323-5cddd39d5667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b01870c-v92xt"
Nov 03 18:09:49.233029 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:49.227340    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfq7g\" (UniqueName: \"kubernetes.io/projected/81f2c5ce-15fa-4010-b323-5cddd39d5667-kube-api-access-vfq7g\") pod \"maas-api-revoke-sub-3b01870c-v92xt\" (UID: \"81f2c5ce-15fa-4010-b323-5cddd39d5667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b01870c-v92xt"
Nov 03 18:09:49.278648 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:49.270679    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b01870c-v92xt"]
Nov 03 18:09:49.425632 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:49.393065    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfq7g\" (UniqueName: \"kubernetes.io/projected/81f2c5ce-15fa-4010-b323-5cddd39d5667-kube-api-access-vfq7g\") pod \"maas-api-revoke-sub-3b01870c-v92xt\" (UID: \"81f2c5ce-15fa-4010-b323-5cddd39d5667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b01870c-v92xt"
Nov 03 18:09:49.425632 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:49.393238    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81f2c5ce-15fa-4010-b323-5cddd39d5667-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b01870c-v92xt\" (UID: \"81f2c5ce-15fa-4010-b323-5cddd39d5667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b01870c-v92xt"
Nov 03 18:09:49.425632 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:49.393370    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81f2c5ce-15fa-4010-b323-5cddd39d5667-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b01870c-v92xt\" (UID: \"81f2c5ce-15fa-4010-b323-5cddd39d5667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b01870c-v92xt"
Nov 03 18:09:49.473394 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:49.473355    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81f2c5ce-15fa-4010-b323-5cddd39d5667-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b01870c-v92xt\" (UID: \"81f2c5ce-15fa-4010-b323-5cddd39d5667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b01870c-v92xt"
Nov 03 18:09:49.537732 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:49.525367    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81f2c5ce-15fa-4010-b323-5cddd39d5667-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b01870c-v92xt\" (UID: \"81f2c5ce-15fa-4010-b323-5cddd39d5667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b01870c-v92xt"
Nov 03 18:09:49.560659 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:49.559492    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfq7g\" (UniqueName: \"kubernetes.io/projected/81f2c5ce-15fa-4010-b323-5cddd39d5667-kube-api-access-vfq7g\") pod \"maas-api-revoke-sub-3b01870c-v92xt\" (UID: \"81f2c5ce-15fa-4010-b323-5cddd39d5667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b01870c-v92xt"
Nov 03 18:09:49.597650 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:49.597299    5315 generic.go:356] "Generic (PLEG): container finished" podID="11671eb9-4824-4e89-9692-b5be9038c549" containerID="12f02d737977cbccd0d09199b884317ef35871693aca2a93336aefa5081dfdb3" exitCode=0
Nov 03 18:09:49.597650 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:49.597484    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q" event={"ID":"11671eb9-4824-4e89-9692-b5be9038c549","Type":"ContainerDied","Data":"12f02d737977cbccd0d09199b884317ef35871693aca2a93336aefa5081dfdb3"}
Nov 03 18:09:49.597650 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:49.597542    5315 scope.go:122] "RemoveContainer" containerID="31303af85f59ce20708ef7dfcdeebc3fe83cf483a50ab05bf91b7020b28cb973"
Nov 03 18:09:49.714056 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:49.714027    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q"
Nov 03 18:09:49.734929 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:49.734901    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b01870c-v92xt"
Nov 03 18:09:49.810682 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:49.810582    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11671eb9-4824-4e89-9692-b5be9038c549-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11671eb9-4824-4e89-9692-b5be9038c549-istio-envoy\") pod \"11671eb9-4824-4e89-9692-b5be9038c549\" (UID: \"11671eb9-4824-4e89-9692-b5be9038c549\") "
Nov 03 18:09:49.810812 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:49.810718    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11671eb9-4824-4e89-9692-b5be9038c549-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11671eb9-4824-4e89-9692-b5be9038c549-istiod-ca-cert\") pod \"11671eb9-4824-4e89-9692-b5be9038c549\" (UID: \"11671eb9-4824-4e89-9692-b5be9038c549\") "
Nov 03 18:09:49.810888 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:49.810869    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11671eb9-4824-4e89-9692-b5be9038c549-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11671eb9-4824-4e89-9692-b5be9038c549-istio-podinfo\") pod \"11671eb9-4824-4e89-9692-b5be9038c549\" (UID: \"11671eb9-4824-4e89-9692-b5be9038c549\") "
Nov 03 18:09:49.810969 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:49.810947    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11671eb9-4824-4e89-9692-b5be9038c549-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11671eb9-4824-4e89-9692-b5be9038c549-workload-certs\") pod \"11671eb9-4824-4e89-9692-b5be9038c549\" (UID: \"11671eb9-4824-4e89-9692-b5be9038c549\") "
Nov 03 18:09:49.811043 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:49.810996    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11671eb9-4824-4e89-9692-b5be9038c549-kube-api-access-qjt66\" (UniqueName: \"kubernetes.io/projected/11671eb9-4824-4e89-9692-b5be9038c549-kube-api-access-qjt66\") pod \"11671eb9-4824-4e89-9692-b5be9038c549\" (UID: \"11671eb9-4824-4e89-9692-b5be9038c549\") "
Nov 03 18:09:49.811043 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:49.811033    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11671eb9-4824-4e89-9692-b5be9038c549-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11671eb9-4824-4e89-9692-b5be9038c549-credential-socket\") pod \"11671eb9-4824-4e89-9692-b5be9038c549\" (UID: \"11671eb9-4824-4e89-9692-b5be9038c549\") "
Nov 03 18:09:49.811168 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:49.811132    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11671eb9-4824-4e89-9692-b5be9038c549-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11671eb9-4824-4e89-9692-b5be9038c549-workload-socket\") pod \"11671eb9-4824-4e89-9692-b5be9038c549\" (UID: \"11671eb9-4824-4e89-9692-b5be9038c549\") "
Nov 03 18:09:49.811283 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:49.811248    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11671eb9-4824-4e89-9692-b5be9038c549-istio-token\" (UniqueName: \"kubernetes.io/projected/11671eb9-4824-4e89-9692-b5be9038c549-istio-token\") pod \"11671eb9-4824-4e89-9692-b5be9038c549\" (UID: \"11671eb9-4824-4e89-9692-b5be9038c549\") "
Nov 03 18:09:49.811412 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:49.811352    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11671eb9-4824-4e89-9692-b5be9038c549-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11671eb9-4824-4e89-9692-b5be9038c549-istio-data\") pod \"11671eb9-4824-4e89-9692-b5be9038c549\" (UID: \"11671eb9-4824-4e89-9692-b5be9038c549\") "
Nov 03 18:09:49.811969 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:49.811721    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11671eb9-4824-4e89-9692-b5be9038c549-credential-socket" pod "11671eb9-4824-4e89-9692-b5be9038c549" (UID: "11671eb9-4824-4e89-9692-b5be9038c549"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:49.811969 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:49.811906    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11671eb9-4824-4e89-9692-b5be9038c549-workload-socket" pod "11671eb9-4824-4e89-9692-b5be9038c549" (UID: "11671eb9-4824-4e89-9692-b5be9038c549"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:49.811969 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:49.811919    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11671eb9-4824-4e89-9692-b5be9038c549-istio-data" pod "11671eb9-4824-4e89-9692-b5be9038c549" (UID: "11671eb9-4824-4e89-9692-b5be9038c549"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:49.811969 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:49.811958    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11671eb9-4824-4e89-9692-b5be9038c549-workload-certs" pod "11671eb9-4824-4e89-9692-b5be9038c549" (UID: "11671eb9-4824-4e89-9692-b5be9038c549"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:49.812260 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:49.812082    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11671eb9-4824-4e89-9692-b5be9038c549-istiod-ca-cert" pod "11671eb9-4824-4e89-9692-b5be9038c549" (UID: "11671eb9-4824-4e89-9692-b5be9038c549"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:09:49.812325 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:49.812279    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11671eb9-4824-4e89-9692-b5be9038c549-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:49.812325 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:49.812292    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11671eb9-4824-4e89-9692-b5be9038c549-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:49.812325 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:49.812302    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11671eb9-4824-4e89-9692-b5be9038c549-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:49.812325 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:49.812311    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11671eb9-4824-4e89-9692-b5be9038c549-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:49.812325 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:49.812320    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11671eb9-4824-4e89-9692-b5be9038c549-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:49.813551 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:49.813525    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11671eb9-4824-4e89-9692-b5be9038c549-istio-envoy" pod "11671eb9-4824-4e89-9692-b5be9038c549" (UID: "11671eb9-4824-4e89-9692-b5be9038c549"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:49.814690 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:49.814661    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11671eb9-4824-4e89-9692-b5be9038c549-istio-podinfo" pod "11671eb9-4824-4e89-9692-b5be9038c549" (UID: "11671eb9-4824-4e89-9692-b5be9038c549"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:09:49.814770 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:49.814683    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11671eb9-4824-4e89-9692-b5be9038c549-istio-token" pod "11671eb9-4824-4e89-9692-b5be9038c549" (UID: "11671eb9-4824-4e89-9692-b5be9038c549"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:09:49.815723 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:49.815695    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11671eb9-4824-4e89-9692-b5be9038c549-kube-api-access-qjt66" pod "11671eb9-4824-4e89-9692-b5be9038c549" (UID: "11671eb9-4824-4e89-9692-b5be9038c549"). InnerVolumeSpecName "kube-api-access-qjt66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:09:49.914325 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:49.914288    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11671eb9-4824-4e89-9692-b5be9038c549-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:49.914325 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:49.914327    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjt66\" (UniqueName: \"kubernetes.io/projected/11671eb9-4824-4e89-9692-b5be9038c549-kube-api-access-qjt66\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:49.914546 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:49.914343    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11671eb9-4824-4e89-9692-b5be9038c549-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:49.914546 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:49.914358    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11671eb9-4824-4e89-9692-b5be9038c549-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:50.114583 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:50.114550    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b01870c-v92xt"]
Nov 03 18:09:50.117828 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:09:50.117794    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f2c5ce_15fa_4010_b323_5cddd39d5667.slice/crio-fbae725961caa64a00fe50f7aec0262f12cc5b34959ef5d36075401fc6ba609d WatchSource:0}: Error finding container fbae725961caa64a00fe50f7aec0262f12cc5b34959ef5d36075401fc6ba609d: Status 404 returned error can't find the container with id fbae725961caa64a00fe50f7aec0262f12cc5b34959ef5d36075401fc6ba609d
Nov 03 18:09:50.607562 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:50.607526    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q"
Nov 03 18:09:50.607793 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:50.607563    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q" event={"ID":"11671eb9-4824-4e89-9692-b5be9038c549","Type":"ContainerDied","Data":"767003902d0a4ff4f427e48e0cd0cd3df919c719b30d41d6fe286c36ae43fb9d"}
Nov 03 18:09:50.607793 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:50.607667    5315 scope.go:122] "RemoveContainer" containerID="12f02d737977cbccd0d09199b884317ef35871693aca2a93336aefa5081dfdb3"
Nov 03 18:09:50.609673 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:50.609581    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b01870c-v92xt" event={"ID":"81f2c5ce-15fa-4010-b323-5cddd39d5667","Type":"ContainerStarted","Data":"9caf8f40e07da83e465cf90633fc665f1f40aa9c9c9b9436ff819cd2a0b6de62"}
Nov 03 18:09:50.609673 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:50.609665    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b01870c-v92xt" event={"ID":"81f2c5ce-15fa-4010-b323-5cddd39d5667","Type":"ContainerStarted","Data":"fbae725961caa64a00fe50f7aec0262f12cc5b34959ef5d36075401fc6ba609d"}
Nov 03 18:09:50.792139 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:50.792105    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w"]
Nov 03 18:09:50.792824 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:50.792809    5315 cpu_manager.go:414] "containerMap: removing container" podUID="11671eb9-4824-4e89-9692-b5be9038c549" containerName="istio-proxy"
Nov 03 18:09:50.792877 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:50.792826    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11671eb9-4824-4e89-9692-b5be9038c549" containerName="istio-proxy"
Nov 03 18:09:50.793000 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:50.792990    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="11671eb9-4824-4e89-9692-b5be9038c549" containerName="istio-proxy"
Nov 03 18:09:50.793036 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:50.793007    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="11671eb9-4824-4e89-9692-b5be9038c549" containerName="istio-proxy"
Nov 03 18:09:50.798224 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:50.798187    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w"
Nov 03 18:09:50.810035 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:50.809983    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b01870c-v92xt" podStartSLOduration=1.8099598860000001 podStartE2EDuration="1.809959886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:09:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:09:50.801537452 +0000 UTC m=+1429.817735522" watchObservedRunningTime="2026-11-03 18:09:50.809959886 +0000 UTC m=+1429.826157958"
Nov 03 18:09:50.856195 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:50.856159    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-b6cb8403-openshift-default-dockercfg-8dxhz\""
Nov 03 18:09:51.148849 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:51.143201    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsdqn\" (UniqueName: \"kubernetes.io/projected/f85c57a2-8cb8-4403-ba55-1de5bf63a921-kube-api-access-xsdqn\") pod \"e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w\" (UID: \"f85c57a2-8cb8-4403-ba55-1de5bf63a921\") " pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w"
Nov 03 18:09:51.291157 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:51.291122    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f85c57a2-8cb8-4403-ba55-1de5bf63a921-istio-podinfo\") pod \"e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w\" (UID: \"f85c57a2-8cb8-4403-ba55-1de5bf63a921\") " pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w"
Nov 03 18:09:51.291435 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:51.291420    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f85c57a2-8cb8-4403-ba55-1de5bf63a921-workload-certs\") pod \"e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w\" (UID: \"f85c57a2-8cb8-4403-ba55-1de5bf63a921\") " pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w"
Nov 03 18:09:51.291543 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:51.291533    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85c57a2-8cb8-4403-ba55-1de5bf63a921-workload-socket\") pod \"e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w\" (UID: \"f85c57a2-8cb8-4403-ba55-1de5bf63a921\") " pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w"
Nov 03 18:09:51.295874 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:51.295841    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f85c57a2-8cb8-4403-ba55-1de5bf63a921-istiod-ca-cert\") pod \"e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w\" (UID: \"f85c57a2-8cb8-4403-ba55-1de5bf63a921\") " pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w"
Nov 03 18:09:51.296188 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:51.296166    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f85c57a2-8cb8-4403-ba55-1de5bf63a921-istio-data\") pod \"e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w\" (UID: \"f85c57a2-8cb8-4403-ba55-1de5bf63a921\") " pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w"
Nov 03 18:09:51.296539 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:51.296307    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85c57a2-8cb8-4403-ba55-1de5bf63a921-credential-socket\") pod \"e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w\" (UID: \"f85c57a2-8cb8-4403-ba55-1de5bf63a921\") " pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w"
Nov 03 18:09:51.296737 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:51.296717    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f85c57a2-8cb8-4403-ba55-1de5bf63a921-istio-envoy\") pod \"e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w\" (UID: \"f85c57a2-8cb8-4403-ba55-1de5bf63a921\") " pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w"
Nov 03 18:09:51.297040 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:51.296870    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f85c57a2-8cb8-4403-ba55-1de5bf63a921-istio-token\") pod \"e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w\" (UID: \"f85c57a2-8cb8-4403-ba55-1de5bf63a921\") " pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w"
Nov 03 18:09:51.398634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:51.388390    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q"]
Nov 03 18:09:51.398634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:51.388447    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-248f8ee3-openshift-default-55db97cd67-k5j7q"]
Nov 03 18:09:51.414917 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:51.400871    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f85c57a2-8cb8-4403-ba55-1de5bf63a921-istiod-ca-cert\") pod \"e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w\" (UID: \"f85c57a2-8cb8-4403-ba55-1de5bf63a921\") " pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w"
Nov 03 18:09:51.414917 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:51.400940    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f85c57a2-8cb8-4403-ba55-1de5bf63a921-istio-data\") pod \"e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w\" (UID: \"f85c57a2-8cb8-4403-ba55-1de5bf63a921\") " pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w"
Nov 03 18:09:51.414917 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:51.400970    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85c57a2-8cb8-4403-ba55-1de5bf63a921-credential-socket\") pod \"e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w\" (UID: \"f85c57a2-8cb8-4403-ba55-1de5bf63a921\") " pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w"
Nov 03 18:09:51.414917 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:51.401012    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f85c57a2-8cb8-4403-ba55-1de5bf63a921-istio-envoy\") pod \"e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w\" (UID: \"f85c57a2-8cb8-4403-ba55-1de5bf63a921\") " pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w"
Nov 03 18:09:51.414917 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:51.401045    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f85c57a2-8cb8-4403-ba55-1de5bf63a921-istio-token\") pod \"e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w\" (UID: \"f85c57a2-8cb8-4403-ba55-1de5bf63a921\") " pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w"
Nov 03 18:09:51.414917 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:51.401246    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsdqn\" (UniqueName: \"kubernetes.io/projected/f85c57a2-8cb8-4403-ba55-1de5bf63a921-kube-api-access-xsdqn\") pod \"e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w\" (UID: \"f85c57a2-8cb8-4403-ba55-1de5bf63a921\") " pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w"
Nov 03 18:09:51.414917 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:51.401319    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f85c57a2-8cb8-4403-ba55-1de5bf63a921-istio-podinfo\") pod \"e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w\" (UID: \"f85c57a2-8cb8-4403-ba55-1de5bf63a921\") " pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w"
Nov 03 18:09:51.414917 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:51.401399    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f85c57a2-8cb8-4403-ba55-1de5bf63a921-workload-certs\") pod \"e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w\" (UID: \"f85c57a2-8cb8-4403-ba55-1de5bf63a921\") " pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w"
Nov 03 18:09:51.414917 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:51.401451    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85c57a2-8cb8-4403-ba55-1de5bf63a921-workload-socket\") pod \"e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w\" (UID: \"f85c57a2-8cb8-4403-ba55-1de5bf63a921\") " pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w"
Nov 03 18:09:51.414917 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:51.402026    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85c57a2-8cb8-4403-ba55-1de5bf63a921-workload-socket\") pod \"e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w\" (UID: \"f85c57a2-8cb8-4403-ba55-1de5bf63a921\") " pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w"
Nov 03 18:09:51.414917 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:51.403353    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f85c57a2-8cb8-4403-ba55-1de5bf63a921-istiod-ca-cert\") pod \"e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w\" (UID: \"f85c57a2-8cb8-4403-ba55-1de5bf63a921\") " pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w"
Nov 03 18:09:51.414917 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:51.403948    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f85c57a2-8cb8-4403-ba55-1de5bf63a921-istio-data\") pod \"e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w\" (UID: \"f85c57a2-8cb8-4403-ba55-1de5bf63a921\") " pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w"
Nov 03 18:09:51.487674 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:51.419326    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85c57a2-8cb8-4403-ba55-1de5bf63a921-credential-socket\") pod \"e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w\" (UID: \"f85c57a2-8cb8-4403-ba55-1de5bf63a921\") " pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w"
Nov 03 18:09:51.487674 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:51.422052    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f85c57a2-8cb8-4403-ba55-1de5bf63a921-workload-certs\") pod \"e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w\" (UID: \"f85c57a2-8cb8-4403-ba55-1de5bf63a921\") " pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w"
Nov 03 18:09:51.487674 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:51.443831    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f85c57a2-8cb8-4403-ba55-1de5bf63a921-istio-podinfo\") pod \"e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w\" (UID: \"f85c57a2-8cb8-4403-ba55-1de5bf63a921\") " pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w"
Nov 03 18:09:51.487674 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:51.470960    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f85c57a2-8cb8-4403-ba55-1de5bf63a921-istio-envoy\") pod \"e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w\" (UID: \"f85c57a2-8cb8-4403-ba55-1de5bf63a921\") " pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w"
Nov 03 18:09:51.561950 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:51.561901    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w"]
Nov 03 18:09:51.577093 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:51.577024    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsdqn\" (UniqueName: \"kubernetes.io/projected/f85c57a2-8cb8-4403-ba55-1de5bf63a921-kube-api-access-xsdqn\") pod \"e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w\" (UID: \"f85c57a2-8cb8-4403-ba55-1de5bf63a921\") " pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w"
Nov 03 18:09:51.581657 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:51.580102    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f85c57a2-8cb8-4403-ba55-1de5bf63a921-istio-token\") pod \"e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w\" (UID: \"f85c57a2-8cb8-4403-ba55-1de5bf63a921\") " pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w"
Nov 03 18:09:51.636470 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:51.636433    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w"
Nov 03 18:09:51.642107 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:51.642081    5315 generic.go:356] "Generic (PLEG): container finished" podID="81f2c5ce-15fa-4010-b323-5cddd39d5667" containerID="9caf8f40e07da83e465cf90633fc665f1f40aa9c9c9b9436ff819cd2a0b6de62" exitCode=0
Nov 03 18:09:51.642239 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:51.642224    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b01870c-v92xt" event={"ID":"81f2c5ce-15fa-4010-b323-5cddd39d5667","Type":"ContainerDied","Data":"9caf8f40e07da83e465cf90633fc665f1f40aa9c9c9b9436ff819cd2a0b6de62"}
Nov 03 18:09:52.089986 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:52.089949    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w"]
Nov 03 18:09:52.092090 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:09:52.092064    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf85c57a2_8cb8_4403_ba55_1de5bf63a921.slice/crio-f39d5c3852ee329e20e20e7638cd38fb7ad176c288235b355d3ee2a063df0bdd WatchSource:0}: Error finding container f39d5c3852ee329e20e20e7638cd38fb7ad176c288235b355d3ee2a063df0bdd: Status 404 returned error can't find the container with id f39d5c3852ee329e20e20e7638cd38fb7ad176c288235b355d3ee2a063df0bdd
Nov 03 18:09:52.094820 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:52.094783    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:09:52.094923 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:52.094863    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:09:52.094923 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:52.094909    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:09:52.653009 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:52.652967    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w" event={"ID":"f85c57a2-8cb8-4403-ba55-1de5bf63a921","Type":"ContainerStarted","Data":"095a31f3667a5bef8c64a5ae510403143a89669654e53377a42100159f0dd681"}
Nov 03 18:09:52.653403 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:52.653035    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w" event={"ID":"f85c57a2-8cb8-4403-ba55-1de5bf63a921","Type":"ContainerStarted","Data":"f39d5c3852ee329e20e20e7638cd38fb7ad176c288235b355d3ee2a063df0bdd"}
Nov 03 18:09:52.773401 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:52.773351    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w" podStartSLOduration=2.7733355939999997 podStartE2EDuration="2.773335594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:09:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:09:52.768065437 +0000 UTC m=+1431.784263510" watchObservedRunningTime="2026-11-03 18:09:52.773335594 +0000 UTC m=+1431.789533706"
Nov 03 18:09:52.813076 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:52.813051    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b01870c-v92xt"
Nov 03 18:09:52.854057 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:52.854030    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81f2c5ce-15fa-4010-b323-5cddd39d5667-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81f2c5ce-15fa-4010-b323-5cddd39d5667-maas-api-xks-ca\") pod \"81f2c5ce-15fa-4010-b323-5cddd39d5667\" (UID: \"81f2c5ce-15fa-4010-b323-5cddd39d5667\") "
Nov 03 18:09:52.854211 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:52.854181    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81f2c5ce-15fa-4010-b323-5cddd39d5667-kube-api-access-vfq7g\" (UniqueName: \"kubernetes.io/projected/81f2c5ce-15fa-4010-b323-5cddd39d5667-kube-api-access-vfq7g\") pod \"81f2c5ce-15fa-4010-b323-5cddd39d5667\" (UID: \"81f2c5ce-15fa-4010-b323-5cddd39d5667\") "
Nov 03 18:09:52.854259 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:52.854217    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81f2c5ce-15fa-4010-b323-5cddd39d5667-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81f2c5ce-15fa-4010-b323-5cddd39d5667-maas-api-openshift-ca\") pod \"81f2c5ce-15fa-4010-b323-5cddd39d5667\" (UID: \"81f2c5ce-15fa-4010-b323-5cddd39d5667\") "
Nov 03 18:09:52.854645 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:52.854600    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81f2c5ce-15fa-4010-b323-5cddd39d5667-maas-api-openshift-ca" pod "81f2c5ce-15fa-4010-b323-5cddd39d5667" (UID: "81f2c5ce-15fa-4010-b323-5cddd39d5667"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:09:52.854814 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:52.854795    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81f2c5ce-15fa-4010-b323-5cddd39d5667-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:52.856274 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:52.856246    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81f2c5ce-15fa-4010-b323-5cddd39d5667-maas-api-xks-ca" pod "81f2c5ce-15fa-4010-b323-5cddd39d5667" (UID: "81f2c5ce-15fa-4010-b323-5cddd39d5667"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:09:52.856365 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:52.856339    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81f2c5ce-15fa-4010-b323-5cddd39d5667-kube-api-access-vfq7g" pod "81f2c5ce-15fa-4010-b323-5cddd39d5667" (UID: "81f2c5ce-15fa-4010-b323-5cddd39d5667"). InnerVolumeSpecName "kube-api-access-vfq7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:09:52.977690 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:52.975420    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81f2c5ce-15fa-4010-b323-5cddd39d5667-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:52.977690 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:52.975463    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfq7g\" (UniqueName: \"kubernetes.io/projected/81f2c5ce-15fa-4010-b323-5cddd39d5667-kube-api-access-vfq7g\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:53.076468 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:53.076420    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c44d52e-ldxbf"]
Nov 03 18:09:53.082952 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:53.082915    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c44d52e-ldxbf"]
Nov 03 18:09:53.111390 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:53.111353    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11671eb9-4824-4e89-9692-b5be9038c549" path="/var/lib/kubelet/pods/11671eb9-4824-4e89-9692-b5be9038c549/volumes"
Nov 03 18:09:53.111992 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:53.111973    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb88a0c1-77c2-495a-bf00-168ce82fa90e" path="/var/lib/kubelet/pods/cb88a0c1-77c2-495a-bf00-168ce82fa90e/volumes"
Nov 03 18:09:53.637833 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:53.637800    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w"
Nov 03 18:09:53.642938 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:53.642908    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w"
Nov 03 18:09:53.663682 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:53.663635    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b01870c-v92xt" event={"ID":"81f2c5ce-15fa-4010-b323-5cddd39d5667","Type":"ContainerDied","Data":"fbae725961caa64a00fe50f7aec0262f12cc5b34959ef5d36075401fc6ba609d"}
Nov 03 18:09:53.663682 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:53.663678    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b01870c-v92xt"
Nov 03 18:09:53.664196 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:53.663692    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbae725961caa64a00fe50f7aec0262f12cc5b34959ef5d36075401fc6ba609d"
Nov 03 18:09:53.664196 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:53.664152    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w"
Nov 03 18:09:53.665001 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:53.664972    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w"
Nov 03 18:09:54.578982 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:09:54.578942    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:09:59.415643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:59.410665    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b6cb8403-77775ddc9f-pmzcv"]
Nov 03 18:09:59.430634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:59.430071    5315 cpu_manager.go:414] "containerMap: removing container" podUID="11671eb9-4824-4e89-9692-b5be9038c549" containerName="istio-proxy"
Nov 03 18:09:59.430634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:59.430143    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11671eb9-4824-4e89-9692-b5be9038c549" containerName="istio-proxy"
Nov 03 18:09:59.430634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:59.430180    5315 cpu_manager.go:414] "containerMap: removing container" podUID="81f2c5ce-15fa-4010-b323-5cddd39d5667" containerName="revoke-keys"
Nov 03 18:09:59.430634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:59.430189    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81f2c5ce-15fa-4010-b323-5cddd39d5667" containerName="revoke-keys"
Nov 03 18:09:59.430634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:59.430451    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="81f2c5ce-15fa-4010-b323-5cddd39d5667" containerName="revoke-keys"
Nov 03 18:09:59.484003 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:59.483968    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b6cb8403-77775ddc9f-pmzcv"]
Nov 03 18:09:59.484330 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:59.484311    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b6cb8403-77775ddc9f-pmzcv"
Nov 03 18:09:59.568543 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:59.558797    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-b6cb8403\""
Nov 03 18:09:59.568543 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:59.564702    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25bc2947-958b-4181-99a7-716108abe703-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b6cb8403-77775ddc9f-pmzcv\" (UID: \"25bc2947-958b-4181-99a7-716108abe703\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b6cb8403-77775ddc9f-pmzcv"
Nov 03 18:09:59.568543 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:59.564828    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltqvh\" (UniqueName: \"kubernetes.io/projected/25bc2947-958b-4181-99a7-716108abe703-kube-api-access-ltqvh\") pod \"payload-pre-processing-e2e-mt-b6cb8403-77775ddc9f-pmzcv\" (UID: \"25bc2947-958b-4181-99a7-716108abe703\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b6cb8403-77775ddc9f-pmzcv"
Nov 03 18:09:59.688071 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:59.684997    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b6cb8403-5545f5d6b7-6m5gp"]
Nov 03 18:09:59.702211 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:59.702154    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25bc2947-958b-4181-99a7-716108abe703-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b6cb8403-77775ddc9f-pmzcv\" (UID: \"25bc2947-958b-4181-99a7-716108abe703\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b6cb8403-77775ddc9f-pmzcv"
Nov 03 18:09:59.740714 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:59.740676    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltqvh\" (UniqueName: \"kubernetes.io/projected/25bc2947-958b-4181-99a7-716108abe703-kube-api-access-ltqvh\") pod \"payload-pre-processing-e2e-mt-b6cb8403-77775ddc9f-pmzcv\" (UID: \"25bc2947-958b-4181-99a7-716108abe703\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b6cb8403-77775ddc9f-pmzcv"
Nov 03 18:09:59.741531 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:59.740394    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25bc2947-958b-4181-99a7-716108abe703-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b6cb8403-77775ddc9f-pmzcv\" (UID: \"25bc2947-958b-4181-99a7-716108abe703\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b6cb8403-77775ddc9f-pmzcv"
Nov 03 18:09:59.968691 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:59.957407    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b6cb8403-5545f5d6b7-6m5gp"
Nov 03 18:09:59.989104 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:09:59.989073    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b6cb8403-5545f5d6b7-6m5gp"]
Nov 03 18:10:00.002507 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:00.002462    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltqvh\" (UniqueName: \"kubernetes.io/projected/25bc2947-958b-4181-99a7-716108abe703-kube-api-access-ltqvh\") pod \"payload-pre-processing-e2e-mt-b6cb8403-77775ddc9f-pmzcv\" (UID: \"25bc2947-958b-4181-99a7-716108abe703\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b6cb8403-77775ddc9f-pmzcv"
Nov 03 18:10:00.037034 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:00.037004    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-b6cb8403-dockercfg-2lzmt\""
Nov 03 18:10:00.049779 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:00.049744    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/581c90bc-60cd-412a-b4d5-5bbab2b49336-plugins-config-volume\") pod \"payload-processing-e2e-mt-b6cb8403-5545f5d6b7-6m5gp\" (UID: \"581c90bc-60cd-412a-b4d5-5bbab2b49336\") " pod="openshift-ingress/payload-processing-e2e-mt-b6cb8403-5545f5d6b7-6m5gp"
Nov 03 18:10:00.049893 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:00.049790    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnfr4\" (UniqueName: \"kubernetes.io/projected/581c90bc-60cd-412a-b4d5-5bbab2b49336-kube-api-access-nnfr4\") pod \"payload-processing-e2e-mt-b6cb8403-5545f5d6b7-6m5gp\" (UID: \"581c90bc-60cd-412a-b4d5-5bbab2b49336\") " pod="openshift-ingress/payload-processing-e2e-mt-b6cb8403-5545f5d6b7-6m5gp"
Nov 03 18:10:00.151394 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:00.151355    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/581c90bc-60cd-412a-b4d5-5bbab2b49336-plugins-config-volume\") pod \"payload-processing-e2e-mt-b6cb8403-5545f5d6b7-6m5gp\" (UID: \"581c90bc-60cd-412a-b4d5-5bbab2b49336\") " pod="openshift-ingress/payload-processing-e2e-mt-b6cb8403-5545f5d6b7-6m5gp"
Nov 03 18:10:00.151394 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:00.151402    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnfr4\" (UniqueName: \"kubernetes.io/projected/581c90bc-60cd-412a-b4d5-5bbab2b49336-kube-api-access-nnfr4\") pod \"payload-processing-e2e-mt-b6cb8403-5545f5d6b7-6m5gp\" (UID: \"581c90bc-60cd-412a-b4d5-5bbab2b49336\") " pod="openshift-ingress/payload-processing-e2e-mt-b6cb8403-5545f5d6b7-6m5gp"
Nov 03 18:10:00.152109 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:00.152085    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/581c90bc-60cd-412a-b4d5-5bbab2b49336-plugins-config-volume\") pod \"payload-processing-e2e-mt-b6cb8403-5545f5d6b7-6m5gp\" (UID: \"581c90bc-60cd-412a-b4d5-5bbab2b49336\") " pod="openshift-ingress/payload-processing-e2e-mt-b6cb8403-5545f5d6b7-6m5gp"
Nov 03 18:10:00.224677 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:00.223732    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnfr4\" (UniqueName: \"kubernetes.io/projected/581c90bc-60cd-412a-b4d5-5bbab2b49336-kube-api-access-nnfr4\") pod \"payload-processing-e2e-mt-b6cb8403-5545f5d6b7-6m5gp\" (UID: \"581c90bc-60cd-412a-b4d5-5bbab2b49336\") " pod="openshift-ingress/payload-processing-e2e-mt-b6cb8403-5545f5d6b7-6m5gp"
Nov 03 18:10:00.287942 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:00.287901    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b6cb8403-77775ddc9f-pmzcv"
Nov 03 18:10:00.354870 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:00.353053    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b6cb8403-5545f5d6b7-6m5gp"
Nov 03 18:10:00.911390 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:00.911363    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b6cb8403-77775ddc9f-pmzcv"]
Nov 03 18:10:00.914489 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:10:00.914451    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25bc2947_958b_4181_99a7_716108abe703.slice/crio-37371535260f2f0afe7f16917151d9f8af253f854289d3a0cf99d5d21807d8f4 WatchSource:0}: Error finding container 37371535260f2f0afe7f16917151d9f8af253f854289d3a0cf99d5d21807d8f4: Status 404 returned error can't find the container with id 37371535260f2f0afe7f16917151d9f8af253f854289d3a0cf99d5d21807d8f4
Nov 03 18:10:00.954637 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:00.954588    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx"]
Nov 03 18:10:00.960899 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:00.960875    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx"
Nov 03 18:10:00.963854 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:00.963829    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b6cb8403-5545f5d6b7-6m5gp"]
Nov 03 18:10:00.968810 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:10:00.968782    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod581c90bc_60cd_412a_b4d5_5bbab2b49336.slice/crio-e871229c62613882bf0b9148ed9bfb52f55c0cd697dd4abde9f97adf82d2ce00 WatchSource:0}: Error finding container e871229c62613882bf0b9148ed9bfb52f55c0cd697dd4abde9f97adf82d2ce00: Status 404 returned error can't find the container with id e871229c62613882bf0b9148ed9bfb52f55c0cd697dd4abde9f97adf82d2ce00
Nov 03 18:10:01.009645 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:01.009592    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-1720c00c-openshift-default-dockercfg-ccffc\""
Nov 03 18:10:01.017238 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:01.017210    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b6cb8403-5545f5d6b7-6m5gp" event={"ID":"581c90bc-60cd-412a-b4d5-5bbab2b49336","Type":"ContainerStarted","Data":"e871229c62613882bf0b9148ed9bfb52f55c0cd697dd4abde9f97adf82d2ce00"}
Nov 03 18:10:01.019546 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:01.019514    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b6cb8403-77775ddc9f-pmzcv" event={"ID":"25bc2947-958b-4181-99a7-716108abe703","Type":"ContainerStarted","Data":"24ed2687b798f034b1d198ae91be48585c7d6e433657c5c342fa984a27fe7d02"}
Nov 03 18:10:01.019674 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:01.019564    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b6cb8403-77775ddc9f-pmzcv" event={"ID":"25bc2947-958b-4181-99a7-716108abe703","Type":"ContainerStarted","Data":"37371535260f2f0afe7f16917151d9f8af253f854289d3a0cf99d5d21807d8f4"}
Nov 03 18:10:01.019674 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:01.019666    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b6cb8403-77775ddc9f-pmzcv"
Nov 03 18:10:01.033943 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:01.033649    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx"]
Nov 03 18:10:01.112359 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:01.112322    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-istio-token\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx\" (UID: \"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx"
Nov 03 18:10:01.112359 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:01.112364    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-istio-envoy\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx\" (UID: \"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx"
Nov 03 18:10:01.112639 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:01.112497    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-istiod-ca-cert\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx\" (UID: \"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx"
Nov 03 18:10:01.112639 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:01.112550    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-workload-certs\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx\" (UID: \"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx"
Nov 03 18:10:01.112639 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:01.112589    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-istio-data\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx\" (UID: \"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx"
Nov 03 18:10:01.112768 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:01.112668    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-credential-socket\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx\" (UID: \"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx"
Nov 03 18:10:01.112768 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:01.112736    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-workload-socket\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx\" (UID: \"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx"
Nov 03 18:10:01.112768 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:01.112760    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmngj\" (UniqueName: \"kubernetes.io/projected/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-kube-api-access-bmngj\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx\" (UID: \"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx"
Nov 03 18:10:01.112925 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:01.112901    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-istio-podinfo\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx\" (UID: \"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx"
Nov 03 18:10:01.229689 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:01.220186    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-b6cb8403-77775ddc9f-pmzcv" podStartSLOduration=2.220164477 podStartE2EDuration="2.220164477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:09:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:10:01.21690571 +0000 UTC m=+1440.233103772" watchObservedRunningTime="2026-11-03 18:10:01.220164477 +0000 UTC m=+1440.236362547"
Nov 03 18:10:01.229689 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:01.225816    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-istio-token\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx\" (UID: \"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx"
Nov 03 18:10:01.229689 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:01.225885    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-istio-envoy\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx\" (UID: \"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx"
Nov 03 18:10:01.229689 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:01.225989    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-istiod-ca-cert\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx\" (UID: \"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx"
Nov 03 18:10:01.229689 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:01.226035    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-workload-certs\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx\" (UID: \"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx"
Nov 03 18:10:01.229689 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:01.226077    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-istio-data\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx\" (UID: \"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx"
Nov 03 18:10:01.229689 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:01.226122    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-credential-socket\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx\" (UID: \"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx"
Nov 03 18:10:01.229689 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:01.226197    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-workload-socket\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx\" (UID: \"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx"
Nov 03 18:10:01.229689 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:01.226228    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmngj\" (UniqueName: \"kubernetes.io/projected/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-kube-api-access-bmngj\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx\" (UID: \"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx"
Nov 03 18:10:01.229689 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:01.226364    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-istio-podinfo\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx\" (UID: \"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx"
Nov 03 18:10:01.229689 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:01.227193    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-workload-certs\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx\" (UID: \"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx"
Nov 03 18:10:01.229689 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:01.227464    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-istio-data\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx\" (UID: \"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx"
Nov 03 18:10:01.229689 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:01.227517    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-istiod-ca-cert\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx\" (UID: \"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx"
Nov 03 18:10:01.229689 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:01.227828    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-credential-socket\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx\" (UID: \"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx"
Nov 03 18:10:01.229689 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:01.228076    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-workload-socket\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx\" (UID: \"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx"
Nov 03 18:10:01.301660 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:01.301587    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-istio-envoy\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx\" (UID: \"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx"
Nov 03 18:10:01.354633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:01.342418    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmngj\" (UniqueName: \"kubernetes.io/projected/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-kube-api-access-bmngj\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx\" (UID: \"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx"
Nov 03 18:10:01.355418 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:01.355371    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-istio-podinfo\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx\" (UID: \"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx"
Nov 03 18:10:01.356712 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:01.355807    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-istio-token\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx\" (UID: \"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx"
Nov 03 18:10:01.629439 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:01.629406    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-1720c00c-openshift-default-dockercfg-ccffc\""
Nov 03 18:10:01.632493 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:01.632465    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx"
Nov 03 18:10:02.030160 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:02.030120    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b6cb8403-5545f5d6b7-6m5gp" event={"ID":"581c90bc-60cd-412a-b4d5-5bbab2b49336","Type":"ContainerStarted","Data":"40d1300622a48f81b2e1a83b63f4b1be759c524180ae9c09713410b6db8814b1"}
Nov 03 18:10:02.030563 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:02.030211    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-b6cb8403-5545f5d6b7-6m5gp"
Nov 03 18:10:02.075295 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:02.075265    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx"]
Nov 03 18:10:02.078666 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:10:02.078591    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ebdce76_7e0e_4916_afcd_2aaf7cc44fed.slice/crio-5fe42f6aa0859da70b8580025d4f7e029430e6ebd4544f864a0ee66fad7a5332 WatchSource:0}: Error finding container 5fe42f6aa0859da70b8580025d4f7e029430e6ebd4544f864a0ee66fad7a5332: Status 404 returned error can't find the container with id 5fe42f6aa0859da70b8580025d4f7e029430e6ebd4544f864a0ee66fad7a5332
Nov 03 18:10:02.081702 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:02.081663    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:10:02.081798 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:02.081742    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:10:02.081798 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:02.081772    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:10:02.141709 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:02.141621    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-b6cb8403-5545f5d6b7-6m5gp" podStartSLOduration=3.141592236 podStartE2EDuration="3.141592236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:09:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:10:02.137661463 +0000 UTC m=+1441.153859532" watchObservedRunningTime="2026-11-03 18:10:02.141592236 +0000 UTC m=+1441.157790304"
Nov 03 18:10:03.041414 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:03.041373    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx" event={"ID":"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed","Type":"ContainerStarted","Data":"6c4ad9e24979e6ced486b18cb044721e965b37d80560cf8ccd1905831306733c"}
Nov 03 18:10:03.041831 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:03.041466    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx" event={"ID":"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed","Type":"ContainerStarted","Data":"5fe42f6aa0859da70b8580025d4f7e029430e6ebd4544f864a0ee66fad7a5332"}
Nov 03 18:10:03.204987 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:03.204924    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx" podStartSLOduration=3.204909936 podStartE2EDuration="3.204909936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:10:03.19966781 +0000 UTC m=+1442.215865880" watchObservedRunningTime="2026-11-03 18:10:03.204909936 +0000 UTC m=+1442.221108004"
Nov 03 18:10:03.633723 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:03.633684    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx"
Nov 03 18:10:03.638896 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:03.638869    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx"
Nov 03 18:10:04.093195 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:04.089106    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx"
Nov 03 18:10:04.108075 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:04.105895    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx"
Nov 03 18:10:09.055549 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:09.055358    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96cc1fde-txnlx"]
Nov 03 18:10:09.146633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:09.143457    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96cc1fde-txnlx"]
Nov 03 18:10:09.892633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:09.886677    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1720c00c-7f49c64bcf-pmktn"]
Nov 03 18:10:09.928633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:09.924026    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1720c00c-7f49c64bcf-pmktn"
Nov 03 18:10:09.937642 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:09.935910    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1720c00c-7f49c64bcf-pmktn"]
Nov 03 18:10:10.065633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:10.062814    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-1720c00c-dockercfg-g9t7d\""
Nov 03 18:10:10.065633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:10.062891    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-1720c00c\""
Nov 03 18:10:10.097835 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:10.097793    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1720c00c-549d8c448c-v5z86"]
Nov 03 18:10:10.104908 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:10.104869    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1720c00c-549d8c448c-v5z86"
Nov 03 18:10:10.125221 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:10.124331    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2ad7926-76d5-48b1-8311-d3adb5742fd2-plugins-config-volume\") pod \"payload-processing-e2e-mt-1720c00c-7f49c64bcf-pmktn\" (UID: \"e2ad7926-76d5-48b1-8311-d3adb5742fd2\") " pod="openshift-ingress/payload-processing-e2e-mt-1720c00c-7f49c64bcf-pmktn"
Nov 03 18:10:10.125221 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:10.124399    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxtlc\" (UniqueName: \"kubernetes.io/projected/e2ad7926-76d5-48b1-8311-d3adb5742fd2-kube-api-access-pxtlc\") pod \"payload-processing-e2e-mt-1720c00c-7f49c64bcf-pmktn\" (UID: \"e2ad7926-76d5-48b1-8311-d3adb5742fd2\") " pod="openshift-ingress/payload-processing-e2e-mt-1720c00c-7f49c64bcf-pmktn"
Nov 03 18:10:10.252095 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:10.250502    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwhcq\" (UniqueName: \"kubernetes.io/projected/c06ba561-05ef-49c8-b410-6aa8cb926b93-kube-api-access-bwhcq\") pod \"payload-pre-processing-e2e-mt-1720c00c-549d8c448c-v5z86\" (UID: \"c06ba561-05ef-49c8-b410-6aa8cb926b93\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1720c00c-549d8c448c-v5z86"
Nov 03 18:10:10.252095 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:10.250569    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c06ba561-05ef-49c8-b410-6aa8cb926b93-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1720c00c-549d8c448c-v5z86\" (UID: \"c06ba561-05ef-49c8-b410-6aa8cb926b93\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1720c00c-549d8c448c-v5z86"
Nov 03 18:10:10.252095 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:10.250763    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2ad7926-76d5-48b1-8311-d3adb5742fd2-plugins-config-volume\") pod \"payload-processing-e2e-mt-1720c00c-7f49c64bcf-pmktn\" (UID: \"e2ad7926-76d5-48b1-8311-d3adb5742fd2\") " pod="openshift-ingress/payload-processing-e2e-mt-1720c00c-7f49c64bcf-pmktn"
Nov 03 18:10:10.252095 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:10.250805    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxtlc\" (UniqueName: \"kubernetes.io/projected/e2ad7926-76d5-48b1-8311-d3adb5742fd2-kube-api-access-pxtlc\") pod \"payload-processing-e2e-mt-1720c00c-7f49c64bcf-pmktn\" (UID: \"e2ad7926-76d5-48b1-8311-d3adb5742fd2\") " pod="openshift-ingress/payload-processing-e2e-mt-1720c00c-7f49c64bcf-pmktn"
Nov 03 18:10:10.252095 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:10.251990    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2ad7926-76d5-48b1-8311-d3adb5742fd2-plugins-config-volume\") pod \"payload-processing-e2e-mt-1720c00c-7f49c64bcf-pmktn\" (UID: \"e2ad7926-76d5-48b1-8311-d3adb5742fd2\") " pod="openshift-ingress/payload-processing-e2e-mt-1720c00c-7f49c64bcf-pmktn"
Nov 03 18:10:10.298011 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:10.297953    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1720c00c-549d8c448c-v5z86"]
Nov 03 18:10:10.393666 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:10.393509    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwhcq\" (UniqueName: \"kubernetes.io/projected/c06ba561-05ef-49c8-b410-6aa8cb926b93-kube-api-access-bwhcq\") pod \"payload-pre-processing-e2e-mt-1720c00c-549d8c448c-v5z86\" (UID: \"c06ba561-05ef-49c8-b410-6aa8cb926b93\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1720c00c-549d8c448c-v5z86"
Nov 03 18:10:10.393666 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:10.393575    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c06ba561-05ef-49c8-b410-6aa8cb926b93-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1720c00c-549d8c448c-v5z86\" (UID: \"c06ba561-05ef-49c8-b410-6aa8cb926b93\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1720c00c-549d8c448c-v5z86"
Nov 03 18:10:10.398661 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:10.394716    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c06ba561-05ef-49c8-b410-6aa8cb926b93-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1720c00c-549d8c448c-v5z86\" (UID: \"c06ba561-05ef-49c8-b410-6aa8cb926b93\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1720c00c-549d8c448c-v5z86"
Nov 03 18:10:10.411298 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:10.411213    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxtlc\" (UniqueName: \"kubernetes.io/projected/e2ad7926-76d5-48b1-8311-d3adb5742fd2-kube-api-access-pxtlc\") pod \"payload-processing-e2e-mt-1720c00c-7f49c64bcf-pmktn\" (UID: \"e2ad7926-76d5-48b1-8311-d3adb5742fd2\") " pod="openshift-ingress/payload-processing-e2e-mt-1720c00c-7f49c64bcf-pmktn"
Nov 03 18:10:10.451046 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:10.451014    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwhcq\" (UniqueName: \"kubernetes.io/projected/c06ba561-05ef-49c8-b410-6aa8cb926b93-kube-api-access-bwhcq\") pod \"payload-pre-processing-e2e-mt-1720c00c-549d8c448c-v5z86\" (UID: \"c06ba561-05ef-49c8-b410-6aa8cb926b93\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1720c00c-549d8c448c-v5z86"
Nov 03 18:10:10.548973 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:10.548940    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1720c00c-549d8c448c-v5z86"
Nov 03 18:10:10.566661 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:10.566589    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1720c00c-7f49c64bcf-pmktn"
Nov 03 18:10:10.950691 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:10.950650    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1720c00c-549d8c448c-v5z86"]
Nov 03 18:10:10.991158 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:10:10.991124    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc06ba561_05ef_49c8_b410_6aa8cb926b93.slice/crio-c585cfa1866edeb9c7194a1bf31f970de585c0c4fb0759baa203b5964106961d WatchSource:0}: Error finding container c585cfa1866edeb9c7194a1bf31f970de585c0c4fb0759baa203b5964106961d: Status 404 returned error can't find the container with id c585cfa1866edeb9c7194a1bf31f970de585c0c4fb0759baa203b5964106961d
Nov 03 18:10:11.002251 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:11.002219    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1720c00c-7f49c64bcf-pmktn"]
Nov 03 18:10:11.006850 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:10:11.006779    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2ad7926_76d5_48b1_8311_d3adb5742fd2.slice/crio-ef25a79792fb289dc3c1d9b08ad77e32351951906e6e037d8adee8bcf72fd133 WatchSource:0}: Error finding container ef25a79792fb289dc3c1d9b08ad77e32351951906e6e037d8adee8bcf72fd133: Status 404 returned error can't find the container with id ef25a79792fb289dc3c1d9b08ad77e32351951906e6e037d8adee8bcf72fd133
Nov 03 18:10:11.106728 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:11.106694    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23d87d88-8b0e-4e14-9fa0-cc704c31e415" path="/var/lib/kubelet/pods/23d87d88-8b0e-4e14-9fa0-cc704c31e415/volumes"
Nov 03 18:10:11.270292 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:11.270186    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1720c00c-549d8c448c-v5z86" event={"ID":"c06ba561-05ef-49c8-b410-6aa8cb926b93","Type":"ContainerStarted","Data":"898f7e8995a9ba67aeaa853d027be769612a9ecbabf900a8efbf59446f782763"}
Nov 03 18:10:11.270292 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:11.270256    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1720c00c-549d8c448c-v5z86" event={"ID":"c06ba561-05ef-49c8-b410-6aa8cb926b93","Type":"ContainerStarted","Data":"c585cfa1866edeb9c7194a1bf31f970de585c0c4fb0759baa203b5964106961d"}
Nov 03 18:10:11.270292 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:11.270289    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-1720c00c-549d8c448c-v5z86"
Nov 03 18:10:11.271933 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:11.271905    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1720c00c-7f49c64bcf-pmktn" event={"ID":"e2ad7926-76d5-48b1-8311-d3adb5742fd2","Type":"ContainerStarted","Data":"0aad1bbb8c55382336f0f49a62296ccd8b182596d6bd82407766e0ba6afe901f"}
Nov 03 18:10:11.272052 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:11.271958    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1720c00c-7f49c64bcf-pmktn" event={"ID":"e2ad7926-76d5-48b1-8311-d3adb5742fd2","Type":"ContainerStarted","Data":"ef25a79792fb289dc3c1d9b08ad77e32351951906e6e037d8adee8bcf72fd133"}
Nov 03 18:10:11.272052 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:11.271990    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-1720c00c-7f49c64bcf-pmktn"
Nov 03 18:10:11.378773 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:11.378719    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-1720c00c-549d8c448c-v5z86" podStartSLOduration=2.378704782 podStartE2EDuration="2.378704782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:10:11.377747727 +0000 UTC m=+1450.393945795" watchObservedRunningTime="2026-11-03 18:10:11.378704782 +0000 UTC m=+1450.394902845"
Nov 03 18:10:11.485597 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:11.485548    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-1720c00c-7f49c64bcf-pmktn" podStartSLOduration=2.485536479 podStartE2EDuration="2.485536479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:10:11.485165446 +0000 UTC m=+1450.501363516" watchObservedRunningTime="2026-11-03 18:10:11.485536479 +0000 UTC m=+1450.501734577"
Nov 03 18:10:12.034025 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:12.033941    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b6cb8403-77775ddc9f-pmzcv"
Nov 03 18:10:12.049870 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:12.049839    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5e4da69-lqlb9"]
Nov 03 18:10:12.096379 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:12.096347    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5e4da69-lqlb9"]
Nov 03 18:10:13.045481 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:13.045450    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-b6cb8403-5545f5d6b7-6m5gp"
Nov 03 18:10:13.107682 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:13.107645    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="806db06a-f8d1-43cd-9a9a-452dbdc0c949" path="/var/lib/kubelet/pods/806db06a-f8d1-43cd-9a9a-452dbdc0c949/volumes"
Nov 03 18:10:19.111810 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:19.111774    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75071396-882t2"]
Nov 03 18:10:19.111810 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:19.111818    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75071396-882t2"]
Nov 03 18:10:20.305786 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:20.305750    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-g99pg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:10:20.306201 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:20.305834    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg" podUID="cb937207-ced4-4180-a554-0b7a86edce58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:10:21.116919 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:21.116867    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d82ee37-166a-4433-9733-1c9f55d6d534" path="/var/lib/kubelet/pods/6d82ee37-166a-4433-9733-1c9f55d6d534/volumes"
Nov 03 18:10:22.288806 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:22.288775    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-1720c00c-549d8c448c-v5z86"
Nov 03 18:10:22.289207 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:22.288842    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-1720c00c-7f49c64bcf-pmktn"
Nov 03 18:10:24.579698 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:10:24.579646    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:10:25.048351 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:25.048321    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ff86cc4-mgz25"]
Nov 03 18:10:25.093976 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:25.093933    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ff86cc4-mgz25"]
Nov 03 18:10:25.107706 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:25.107675    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e" path="/var/lib/kubelet/pods/c23e6e4d-f0f9-4fc9-8d09-d2e9a9a5503e/volumes"
Nov 03 18:10:26.059534 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:26.059502    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j"]
Nov 03 18:10:26.065147 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:26.065127    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j"
Nov 03 18:10:26.154030 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:26.153995    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-6pq2j\" (UID: \"205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j"
Nov 03 18:10:26.154222 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:26.154120    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-6pq2j\" (UID: \"205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j"
Nov 03 18:10:26.154222 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:26.154194    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p2vg\" (UniqueName: \"kubernetes.io/projected/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-kube-api-access-5p2vg\") pod \"maas-default-gateway-openshift-default-6dcf757899-6pq2j\" (UID: \"205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j"
Nov 03 18:10:26.154339 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:26.154284    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-6pq2j\" (UID: \"205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j"
Nov 03 18:10:26.154376 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:26.154337    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-6pq2j\" (UID: \"205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j"
Nov 03 18:10:26.154413 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:26.154373    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-6pq2j\" (UID: \"205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j"
Nov 03 18:10:26.154461 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:26.154412    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-6pq2j\" (UID: \"205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j"
Nov 03 18:10:26.154536 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:26.154519    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-6pq2j\" (UID: \"205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j"
Nov 03 18:10:26.154695 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:26.154679    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-6pq2j\" (UID: \"205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j"
Nov 03 18:10:26.173672 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:26.173647    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j"]
Nov 03 18:10:26.256023 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:26.255985    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-6pq2j\" (UID: \"205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j"
Nov 03 18:10:26.256227 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:26.256065    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-6pq2j\" (UID: \"205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j"
Nov 03 18:10:26.256227 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:26.256139    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-6pq2j\" (UID: \"205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j"
Nov 03 18:10:26.256227 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:26.256212    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p2vg\" (UniqueName: \"kubernetes.io/projected/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-kube-api-access-5p2vg\") pod \"maas-default-gateway-openshift-default-6dcf757899-6pq2j\" (UID: \"205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j"
Nov 03 18:10:26.256412 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:26.256287    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-6pq2j\" (UID: \"205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j"
Nov 03 18:10:26.256412 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:26.256335    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-6pq2j\" (UID: \"205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j"
Nov 03 18:10:26.256412 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:26.256385    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-6pq2j\" (UID: \"205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j"
Nov 03 18:10:26.256581 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:26.256442    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-6pq2j\" (UID: \"205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j"
Nov 03 18:10:26.256581 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:26.256490    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-6pq2j\" (UID: \"205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j"
Nov 03 18:10:26.256581 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:26.256532    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-6pq2j\" (UID: \"205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j"
Nov 03 18:10:26.256822 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:26.256737    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-6pq2j\" (UID: \"205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j"
Nov 03 18:10:26.256957 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:26.256932    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-6pq2j\" (UID: \"205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j"
Nov 03 18:10:26.257070 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:26.257038    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-6pq2j\" (UID: \"205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j"
Nov 03 18:10:26.257429 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:26.257405    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-6pq2j\" (UID: \"205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j"
Nov 03 18:10:26.258597 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:26.258576    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-6pq2j\" (UID: \"205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j"
Nov 03 18:10:26.258957 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:26.258935    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-6pq2j\" (UID: \"205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j"
Nov 03 18:10:26.314207 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:26.314128    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-6pq2j\" (UID: \"205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j"
Nov 03 18:10:26.314334 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:26.314289    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p2vg\" (UniqueName: \"kubernetes.io/projected/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-kube-api-access-5p2vg\") pod \"maas-default-gateway-openshift-default-6dcf757899-6pq2j\" (UID: \"205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j"
Nov 03 18:10:26.381226 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:26.381190    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j"
Nov 03 18:10:26.817311 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:26.817269    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j"]
Nov 03 18:10:26.820659 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:10:26.820596    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod205e40fe_2a6e_4e87_aad5_a2ee6b2dfbcc.slice/crio-0010b07117fc436e0d949e411fc54e4733a3b607c9a28b207c77ee2ada2b58e6 WatchSource:0}: Error finding container 0010b07117fc436e0d949e411fc54e4733a3b607c9a28b207c77ee2ada2b58e6: Status 404 returned error can't find the container with id 0010b07117fc436e0d949e411fc54e4733a3b607c9a28b207c77ee2ada2b58e6
Nov 03 18:10:26.823351 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:26.823316    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:10:26.823448 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:26.823392    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:10:26.823448 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:26.823427    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:10:27.509542 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:27.509496    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j" event={"ID":"205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc","Type":"ContainerStarted","Data":"22a056e2c7fd07e28d24035fd8cfb51f6d6907ec08b2f74009bcf3608a4c7a95"}
Nov 03 18:10:27.510173 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:27.509568    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j" event={"ID":"205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc","Type":"ContainerStarted","Data":"0010b07117fc436e0d949e411fc54e4733a3b607c9a28b207c77ee2ada2b58e6"}
Nov 03 18:10:28.381462 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:28.381422    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j"
Nov 03 18:10:28.383299 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:28.383266    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-6pq2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 03 18:10:28.383444 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:28.383318    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j" podUID="205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 03 18:10:29.382494 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:29.382460    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-6pq2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 03 18:10:29.382937 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:29.382515    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j" podUID="205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 03 18:10:30.381985 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:30.381942    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-6pq2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 03 18:10:30.382169 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:30.382004    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j" podUID="205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 03 18:10:31.382312 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:31.382277    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-6pq2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 03 18:10:31.382783 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:31.382336    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j" podUID="205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 03 18:10:32.382380 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:32.382336    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-6pq2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 03 18:10:32.382926 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:32.382407    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j" podUID="205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 03 18:10:33.381868 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:33.381817    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-6pq2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 03 18:10:33.382088 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:33.381893    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j" podUID="205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 03 18:10:34.059317 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:34.059252    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j" podStartSLOduration=8.059234219 podStartE2EDuration="8.059234219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:10:27.643968514 +0000 UTC m=+1466.660166625" watchObservedRunningTime="2026-11-03 18:10:34.059234219 +0000 UTC m=+1473.075432289"
Nov 03 18:10:34.061232 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:34.061205    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45755c76-x67jk"]
Nov 03 18:10:34.106336 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:34.106304    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45755c76-x67jk"]
Nov 03 18:10:34.381672 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:34.381577    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-6pq2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 03 18:10:34.381672 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:34.381647    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j" podUID="205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 03 18:10:35.053300 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:35.053255    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a83934d6-nzswt"]
Nov 03 18:10:35.095164 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:35.095127    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a83934d6-nzswt"]
Nov 03 18:10:35.110538 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:35.110508    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ddbe99a-8494-4a0e-bded-a55c191297f3" path="/var/lib/kubelet/pods/0ddbe99a-8494-4a0e-bded-a55c191297f3/volumes"
Nov 03 18:10:35.111062 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:35.111045    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b23d8e6-0f73-4545-bbdf-e003b811018f" path="/var/lib/kubelet/pods/2b23d8e6-0f73-4545-bbdf-e003b811018f/volumes"
Nov 03 18:10:35.386693 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:35.386592    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j"
Nov 03 18:10:35.386981 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:35.386960    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j"
Nov 03 18:10:35.387553 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:35.387534    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j"
Nov 03 18:10:49.048031 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:49.047989    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2d14a24-fw975"]
Nov 03 18:10:49.093856 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:49.093825    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2d14a24-fw975"]
Nov 03 18:10:49.106988 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:49.106952    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a91be802-1c0f-4e1a-8780-261497c1b71d" path="/var/lib/kubelet/pods/a91be802-1c0f-4e1a-8780-261497c1b71d/volumes"
Nov 03 18:10:53.048882 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:53.048843    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fdfba9b-x4wnb"]
Nov 03 18:10:53.118620 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:53.118585    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7995a32-gl2x5"]
Nov 03 18:10:53.118792 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:53.118645    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fdfba9b-x4wnb"]
Nov 03 18:10:53.118792 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:53.118666    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7995a32-gl2x5"]
Nov 03 18:10:54.584763 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:10:54.584677    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:10:55.060805 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:55.060759    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37d7625f-rqgpz"]
Nov 03 18:10:55.109992 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:55.109948    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca2b4231-a18a-4290-afee-852f0db90ea4" path="/var/lib/kubelet/pods/ca2b4231-a18a-4290-afee-852f0db90ea4/volumes"
Nov 03 18:10:55.110731 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:55.110708    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd07730f-0cf5-4a0a-b21e-e9b96dd7942a" path="/var/lib/kubelet/pods/fd07730f-0cf5-4a0a-b21e-e9b96dd7942a/volumes"
Nov 03 18:10:55.111198 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:55.111178    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37d7625f-rqgpz"]
Nov 03 18:10:57.056361 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:57.056321    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2610a6e1-9nwl5"]
Nov 03 18:10:57.110890 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:57.110856    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e598e385-1a33-43c4-846b-8cc4e29f138e" path="/var/lib/kubelet/pods/e598e385-1a33-43c4-846b-8cc4e29f138e/volumes"
Nov 03 18:10:57.111227 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:57.111211    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b75ece4b-f94kw"]
Nov 03 18:10:57.111284 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:57.111243    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2610a6e1-9nwl5"]
Nov 03 18:10:57.111284 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:57.111263    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b75ece4b-f94kw"]
Nov 03 18:10:59.165759 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:59.165695    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="510cff64-6154-42a0-8657-5a99aa556133" path="/var/lib/kubelet/pods/510cff64-6154-42a0-8657-5a99aa556133/volumes"
Nov 03 18:10:59.166669 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:59.166634    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d33f7de6-f94f-4d92-8832-5bddc50358e4" path="/var/lib/kubelet/pods/d33f7de6-f94f-4d92-8832-5bddc50358e4/volumes"
Nov 03 18:10:59.179015 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:59.178983    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de9baa93-n7c5d"]
Nov 03 18:10:59.220212 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:10:59.220174    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de9baa93-n7c5d"]
Nov 03 18:11:01.110853 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:01.110815    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32d4f2d5-1cc6-438d-b734-00ba0ff2d187" path="/var/lib/kubelet/pods/32d4f2d5-1cc6-438d-b734-00ba0ff2d187/volumes"
Nov 03 18:11:01.534766 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:01.534728    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-97p65_c29d200a-c917-46e5-9cf5-21f4d952e470/ovn-acl-logging/0.log"
Nov 03 18:11:01.571229 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:01.571202    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-97p65_c29d200a-c917-46e5-9cf5-21f4d952e470/ovn-acl-logging/0.log"
Nov 03 18:11:02.511137 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:02.511100    5315 scope.go:122] "RemoveContainer" containerID="9967728e3521f5dbfc2fe4388675b4dc7fbcea9b33967bcb4cbffa9d71b46d70"
Nov 03 18:11:04.773640 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:04.773565    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d"]
Nov 03 18:11:04.798591 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:04.796808    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d"
Nov 03 18:11:04.852635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:04.848037    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-260dd0-openshift-default-dockercfg-2qkbq\""
Nov 03 18:11:05.171941 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:05.170408    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d"]
Nov 03 18:11:05.211513 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:05.211470    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-istiod-ca-cert\") pod \"e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d\" (UID: \"ae6e0c1a-a26e-4b44-8e94-540944db8bd8\") " pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d"
Nov 03 18:11:05.211911 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:05.211886    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-workload-certs\") pod \"e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d\" (UID: \"ae6e0c1a-a26e-4b44-8e94-540944db8bd8\") " pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d"
Nov 03 18:11:05.212134 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:05.212113    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-workload-socket\") pod \"e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d\" (UID: \"ae6e0c1a-a26e-4b44-8e94-540944db8bd8\") " pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d"
Nov 03 18:11:05.212271 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:05.212252    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-istio-data\") pod \"e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d\" (UID: \"ae6e0c1a-a26e-4b44-8e94-540944db8bd8\") " pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d"
Nov 03 18:11:05.212518 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:05.212485    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-istio-token\") pod \"e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d\" (UID: \"ae6e0c1a-a26e-4b44-8e94-540944db8bd8\") " pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d"
Nov 03 18:11:05.212725 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:05.212702    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8hvh\" (UniqueName: \"kubernetes.io/projected/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-kube-api-access-v8hvh\") pod \"e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d\" (UID: \"ae6e0c1a-a26e-4b44-8e94-540944db8bd8\") " pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d"
Nov 03 18:11:05.213008 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:05.212985    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-credential-socket\") pod \"e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d\" (UID: \"ae6e0c1a-a26e-4b44-8e94-540944db8bd8\") " pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d"
Nov 03 18:11:05.213153 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:05.213131    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-istio-podinfo\") pod \"e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d\" (UID: \"ae6e0c1a-a26e-4b44-8e94-540944db8bd8\") " pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d"
Nov 03 18:11:05.213379 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:05.213357    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-istio-envoy\") pod \"e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d\" (UID: \"ae6e0c1a-a26e-4b44-8e94-540944db8bd8\") " pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d"
Nov 03 18:11:05.431468 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:05.431378    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-workload-socket\") pod \"e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d\" (UID: \"ae6e0c1a-a26e-4b44-8e94-540944db8bd8\") " pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d"
Nov 03 18:11:05.431799 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:05.431770    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-istio-data\") pod \"e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d\" (UID: \"ae6e0c1a-a26e-4b44-8e94-540944db8bd8\") " pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d"
Nov 03 18:11:05.432090 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:05.432065    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-istio-token\") pod \"e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d\" (UID: \"ae6e0c1a-a26e-4b44-8e94-540944db8bd8\") " pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d"
Nov 03 18:11:05.433279 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:05.433258    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8hvh\" (UniqueName: \"kubernetes.io/projected/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-kube-api-access-v8hvh\") pod \"e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d\" (UID: \"ae6e0c1a-a26e-4b44-8e94-540944db8bd8\") " pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d"
Nov 03 18:11:05.433566 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:05.433547    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-credential-socket\") pod \"e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d\" (UID: \"ae6e0c1a-a26e-4b44-8e94-540944db8bd8\") " pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d"
Nov 03 18:11:05.434425 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:05.434404    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-istio-podinfo\") pod \"e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d\" (UID: \"ae6e0c1a-a26e-4b44-8e94-540944db8bd8\") " pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d"
Nov 03 18:11:05.435062 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:05.435043    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-istio-envoy\") pod \"e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d\" (UID: \"ae6e0c1a-a26e-4b44-8e94-540944db8bd8\") " pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d"
Nov 03 18:11:05.435874 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:05.435849    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-istiod-ca-cert\") pod \"e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d\" (UID: \"ae6e0c1a-a26e-4b44-8e94-540944db8bd8\") " pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d"
Nov 03 18:11:05.436012 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:05.435994    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-workload-certs\") pod \"e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d\" (UID: \"ae6e0c1a-a26e-4b44-8e94-540944db8bd8\") " pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d"
Nov 03 18:11:05.436235 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:05.433830    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-credential-socket\") pod \"e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d\" (UID: \"ae6e0c1a-a26e-4b44-8e94-540944db8bd8\") " pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d"
Nov 03 18:11:05.436307 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:05.433866    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-istio-data\") pod \"e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d\" (UID: \"ae6e0c1a-a26e-4b44-8e94-540944db8bd8\") " pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d"
Nov 03 18:11:05.437262 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:05.437196    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-istiod-ca-cert\") pod \"e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d\" (UID: \"ae6e0c1a-a26e-4b44-8e94-540944db8bd8\") " pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d"
Nov 03 18:11:05.437478 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:05.433731    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-workload-socket\") pod \"e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d\" (UID: \"ae6e0c1a-a26e-4b44-8e94-540944db8bd8\") " pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d"
Nov 03 18:11:05.437737 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:05.437712    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-workload-certs\") pod \"e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d\" (UID: \"ae6e0c1a-a26e-4b44-8e94-540944db8bd8\") " pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d"
Nov 03 18:11:05.446444 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:05.446391    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-istio-podinfo\") pod \"e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d\" (UID: \"ae6e0c1a-a26e-4b44-8e94-540944db8bd8\") " pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d"
Nov 03 18:11:05.447158 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:05.447116    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-istio-envoy\") pod \"e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d\" (UID: \"ae6e0c1a-a26e-4b44-8e94-540944db8bd8\") " pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d"
Nov 03 18:11:05.491905 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:05.491844    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-istio-token\") pod \"e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d\" (UID: \"ae6e0c1a-a26e-4b44-8e94-540944db8bd8\") " pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d"
Nov 03 18:11:05.491905 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:05.491887    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8hvh\" (UniqueName: \"kubernetes.io/projected/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-kube-api-access-v8hvh\") pod \"e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d\" (UID: \"ae6e0c1a-a26e-4b44-8e94-540944db8bd8\") " pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d"
Nov 03 18:11:05.514632 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:05.513817    5315 patch_prober.go:29] interesting pod/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:11:05.514632 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:05.513906    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67" podUID="d7899300-736f-4a03-aef7-253d7a1b1e2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:11:05.763016 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:05.762915    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d"
Nov 03 18:11:06.318883 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:06.261114    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d"]
Nov 03 18:11:06.318883 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:11:06.264474    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae6e0c1a_a26e_4b44_8e94_540944db8bd8.slice/crio-9ecc1b9db41027fde2bf399d9305ac94dee59bc9c6089161f8ccc872050f4ad0 WatchSource:0}: Error finding container 9ecc1b9db41027fde2bf399d9305ac94dee59bc9c6089161f8ccc872050f4ad0: Status 404 returned error can't find the container with id 9ecc1b9db41027fde2bf399d9305ac94dee59bc9c6089161f8ccc872050f4ad0
Nov 03 18:11:06.318883 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:06.267997    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:11:06.318883 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:06.268066    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:11:06.318883 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:06.268095    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:11:06.471601 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:06.465911    5315 generic.go:356] "Generic (PLEG): container finished" podID="38acc846-2251-4000-ae98-e5a2492da6a6" containerID="4f68f393ea3076a8de212d5b8c368a7c2cfd504c1693f3bbf8e150c268a96686" exitCode=0
Nov 03 18:11:06.471601 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:06.466075    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c" event={"ID":"38acc846-2251-4000-ae98-e5a2492da6a6","Type":"ContainerDied","Data":"4f68f393ea3076a8de212d5b8c368a7c2cfd504c1693f3bbf8e150c268a96686"}
Nov 03 18:11:06.471601 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:06.466638    5315 scope.go:122] "RemoveContainer" containerID="4f68f393ea3076a8de212d5b8c368a7c2cfd504c1693f3bbf8e150c268a96686"
Nov 03 18:11:06.471601 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:06.469439    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:11:06.471601 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:06.469516    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:11:06.471601 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:06.469552    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:11:06.471601 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:06.469752    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d" event={"ID":"ae6e0c1a-a26e-4b44-8e94-540944db8bd8","Type":"ContainerStarted","Data":"15b420fad5c451d5c774e7dea258301c1281506c0f5e05beb989a445f9b6ca1f"}
Nov 03 18:11:06.471601 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:06.469813    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d" event={"ID":"ae6e0c1a-a26e-4b44-8e94-540944db8bd8","Type":"ContainerStarted","Data":"9ecc1b9db41027fde2bf399d9305ac94dee59bc9c6089161f8ccc872050f4ad0"}
Nov 03 18:11:06.722811 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:06.722767    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c"
Nov 03 18:11:06.726812 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:06.726745    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d" podStartSLOduration=2.726723859 podStartE2EDuration="2.726723859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:11:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:11:06.700604418 +0000 UTC m=+1505.716802486" watchObservedRunningTime="2026-11-03 18:11:06.726723859 +0000 UTC m=+1505.742921930"
Nov 03 18:11:06.763660 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:06.763594    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d"
Nov 03 18:11:07.051075 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:07.051041    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc80bea1-xm6fj"]
Nov 03 18:11:07.056099 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:07.056069    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc80bea1-xm6fj"]
Nov 03 18:11:07.108554 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:07.108515    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9a18cf4-7b26-47b3-8295-eb63eaa924d1" path="/var/lib/kubelet/pods/c9a18cf4-7b26-47b3-8295-eb63eaa924d1/volumes"
Nov 03 18:11:07.485714 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:07.483906    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c" event={"ID":"38acc846-2251-4000-ae98-e5a2492da6a6","Type":"ContainerStarted","Data":"3445112f581194bfc56d5fb23a22b94eff030673ff239c6b2118b93cdb1d9863"}
Nov 03 18:11:07.705863 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:07.705823    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c"
Nov 03 18:11:07.706945 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:07.706918    5315 patch_prober.go:29] interesting pod/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 03 18:11:07.707077 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:07.706975    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c" podUID="38acc846-2251-4000-ae98-e5a2492da6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 03 18:11:07.769475 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:07.769394    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d"
Nov 03 18:11:08.493028 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:08.492985    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d"
Nov 03 18:11:08.494033 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:08.494010    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d"
Nov 03 18:11:08.711731 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:08.708713    5315 patch_prober.go:29] interesting pod/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 03 18:11:08.711731 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:08.708781    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c" podUID="38acc846-2251-4000-ae98-e5a2492da6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 03 18:11:09.706004 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:09.705965    5315 patch_prober.go:29] interesting pod/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 03 18:11:09.720801 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:09.706031    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c" podUID="38acc846-2251-4000-ae98-e5a2492da6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 03 18:11:10.512316 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:10.512283    5315 generic.go:356] "Generic (PLEG): container finished" podID="d7899300-736f-4a03-aef7-253d7a1b1e2f" containerID="750e09020663ab70c545beeced3a22c7d61ce9e5431603a6d98aa87a8eeb4264" exitCode=0
Nov 03 18:11:10.512491 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:10.512356    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67" event={"ID":"d7899300-736f-4a03-aef7-253d7a1b1e2f","Type":"ContainerDied","Data":"750e09020663ab70c545beeced3a22c7d61ce9e5431603a6d98aa87a8eeb4264"}
Nov 03 18:11:10.512962 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:10.512940    5315 scope.go:122] "RemoveContainer" containerID="750e09020663ab70c545beeced3a22c7d61ce9e5431603a6d98aa87a8eeb4264"
Nov 03 18:11:10.514591 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:10.514564    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:11:10.514677 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:10.514649    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:11:10.514716 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:10.514685    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:11:10.706970 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:10.706910    5315 patch_prober.go:29] interesting pod/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 03 18:11:10.707443 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:10.706971    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c" podUID="38acc846-2251-4000-ae98-e5a2492da6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 03 18:11:11.523333 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:11.523298    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67" event={"ID":"d7899300-736f-4a03-aef7-253d7a1b1e2f","Type":"ContainerStarted","Data":"ee4cb352ab64ce12813538b6442b423954bd6f3bb1ac379a481de909af752543"}
Nov 03 18:11:11.673989 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:11.673943    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67"
Nov 03 18:11:11.674985 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:11.674930    5315 patch_prober.go:29] interesting pod/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 03 18:11:11.676571 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:11.676522    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67" podUID="d7899300-736f-4a03-aef7-253d7a1b1e2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 03 18:11:11.705851 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:11.705813    5315 patch_prober.go:29] interesting pod/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 03 18:11:11.706029 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:11.705867    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c" podUID="38acc846-2251-4000-ae98-e5a2492da6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 03 18:11:12.667635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:12.663063    5315 patch_prober.go:29] interesting pod/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 03 18:11:12.667635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:12.663149    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67" podUID="d7899300-736f-4a03-aef7-253d7a1b1e2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 03 18:11:12.705912 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:12.705867    5315 patch_prober.go:29] interesting pod/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 03 18:11:12.706095 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:12.705938    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c" podUID="38acc846-2251-4000-ae98-e5a2492da6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 03 18:11:13.072349 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:13.072304    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07f3d541-btzqt"]
Nov 03 18:11:13.094926 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:13.094889    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07f3d541-btzqt"]
Nov 03 18:11:13.107085 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:13.107050    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d" path="/var/lib/kubelet/pods/ac972aa2-f7a1-435f-8fd7-4814ab5e0f0d/volumes"
Nov 03 18:11:13.682630 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:13.674699    5315 patch_prober.go:29] interesting pod/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 03 18:11:13.682630 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:13.674778    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67" podUID="d7899300-736f-4a03-aef7-253d7a1b1e2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 03 18:11:13.879161 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:13.879109    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c"
Nov 03 18:11:13.879672 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:13.879647    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c"
Nov 03 18:11:13.880471 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:13.880450    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c"
Nov 03 18:11:14.661558 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:14.661515    5315 patch_prober.go:29] interesting pod/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 03 18:11:14.661776 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:14.661582    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67" podUID="d7899300-736f-4a03-aef7-253d7a1b1e2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 03 18:11:15.795648 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:15.794947    5315 patch_prober.go:29] interesting pod/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 03 18:11:15.795648 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:15.795027    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67" podUID="d7899300-736f-4a03-aef7-253d7a1b1e2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 03 18:11:15.969674 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:15.964439    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ffd208e-bkvnb"]
Nov 03 18:11:16.053697 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:16.047785    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ffd208e-bkvnb"]
Nov 03 18:11:16.053697 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:16.048263    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ffd208e-bkvnb"
Nov 03 18:11:16.112798 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:16.104239    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-260dd0-76766ff95-8999v"]
Nov 03 18:11:16.112798 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:16.109536    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-260dd0-76766ff95-8999v"
Nov 03 18:11:16.114689 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:16.114663    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-260dd0-76766ff95-8999v"]
Nov 03 18:11:16.119222 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:16.119194    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:11:16.119395 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:16.119287    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-260dd0-5469bdccd7-82scv"]
Nov 03 18:11:16.129879 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:16.129851    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-260dd0-5469bdccd7-82scv"
Nov 03 18:11:16.158963 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:16.158930    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-260dd0-5469bdccd7-82scv"]
Nov 03 18:11:16.165802 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:16.165773    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-260dd0\""
Nov 03 18:11:16.165974 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:16.165837    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:11:16.165974 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:16.165915    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:11:16.166317 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:16.166292    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:11:16.166525 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:16.166506    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-260dd0-dockercfg-gwzzt\""
Nov 03 18:11:16.171263 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:16.171238    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqxn4\" (UniqueName: \"kubernetes.io/projected/05817bd1-81be-4585-b1c2-0f3ca80eec9d-kube-api-access-tqxn4\") pod \"payload-pre-processing-e2e-shared-b-w4-260dd0-5469bdccd7-82scv\" (UID: \"05817bd1-81be-4585-b1c2-0f3ca80eec9d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-260dd0-5469bdccd7-82scv"
Nov 03 18:11:16.171402 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:16.171358    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18f47428-da97-456a-8200-5f665c9f5d46-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-5ffd208e-bkvnb\" (UID: \"18f47428-da97-456a-8200-5f665c9f5d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ffd208e-bkvnb"
Nov 03 18:11:16.172163 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:16.171506    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bshlb\" (UniqueName: \"kubernetes.io/projected/232bbb8c-f8b8-42a7-a29c-f288fe7e99f9-kube-api-access-bshlb\") pod \"payload-processing-e2e-shared-b-w4-260dd0-76766ff95-8999v\" (UID: \"232bbb8c-f8b8-42a7-a29c-f288fe7e99f9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-260dd0-76766ff95-8999v"
Nov 03 18:11:16.172272 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:16.172217    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05817bd1-81be-4585-b1c2-0f3ca80eec9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-260dd0-5469bdccd7-82scv\" (UID: \"05817bd1-81be-4585-b1c2-0f3ca80eec9d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-260dd0-5469bdccd7-82scv"
Nov 03 18:11:16.172342 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:16.172325    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltzwc\" (UniqueName: \"kubernetes.io/projected/18f47428-da97-456a-8200-5f665c9f5d46-kube-api-access-ltzwc\") pod \"maas-api-revoke-keys-e2e-derive-5ffd208e-bkvnb\" (UID: \"18f47428-da97-456a-8200-5f665c9f5d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ffd208e-bkvnb"
Nov 03 18:11:16.172987 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:16.172969    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/232bbb8c-f8b8-42a7-a29c-f288fe7e99f9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-260dd0-76766ff95-8999v\" (UID: \"232bbb8c-f8b8-42a7-a29c-f288fe7e99f9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-260dd0-76766ff95-8999v"
Nov 03 18:11:16.173091 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:16.173009    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18f47428-da97-456a-8200-5f665c9f5d46-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-5ffd208e-bkvnb\" (UID: \"18f47428-da97-456a-8200-5f665c9f5d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ffd208e-bkvnb"
Nov 03 18:11:16.274986 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:16.274948    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqxn4\" (UniqueName: \"kubernetes.io/projected/05817bd1-81be-4585-b1c2-0f3ca80eec9d-kube-api-access-tqxn4\") pod \"payload-pre-processing-e2e-shared-b-w4-260dd0-5469bdccd7-82scv\" (UID: \"05817bd1-81be-4585-b1c2-0f3ca80eec9d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-260dd0-5469bdccd7-82scv"
Nov 03 18:11:16.275210 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:16.275059    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18f47428-da97-456a-8200-5f665c9f5d46-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-5ffd208e-bkvnb\" (UID: \"18f47428-da97-456a-8200-5f665c9f5d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ffd208e-bkvnb"
Nov 03 18:11:16.275210 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:16.275175    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bshlb\" (UniqueName: \"kubernetes.io/projected/232bbb8c-f8b8-42a7-a29c-f288fe7e99f9-kube-api-access-bshlb\") pod \"payload-processing-e2e-shared-b-w4-260dd0-76766ff95-8999v\" (UID: \"232bbb8c-f8b8-42a7-a29c-f288fe7e99f9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-260dd0-76766ff95-8999v"
Nov 03 18:11:16.275336 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:16.275231    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05817bd1-81be-4585-b1c2-0f3ca80eec9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-260dd0-5469bdccd7-82scv\" (UID: \"05817bd1-81be-4585-b1c2-0f3ca80eec9d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-260dd0-5469bdccd7-82scv"
Nov 03 18:11:16.275336 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:16.275295    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltzwc\" (UniqueName: \"kubernetes.io/projected/18f47428-da97-456a-8200-5f665c9f5d46-kube-api-access-ltzwc\") pod \"maas-api-revoke-keys-e2e-derive-5ffd208e-bkvnb\" (UID: \"18f47428-da97-456a-8200-5f665c9f5d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ffd208e-bkvnb"
Nov 03 18:11:16.275421 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:16.275347    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/232bbb8c-f8b8-42a7-a29c-f288fe7e99f9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-260dd0-76766ff95-8999v\" (UID: \"232bbb8c-f8b8-42a7-a29c-f288fe7e99f9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-260dd0-76766ff95-8999v"
Nov 03 18:11:16.275421 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:16.275386    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18f47428-da97-456a-8200-5f665c9f5d46-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-5ffd208e-bkvnb\" (UID: \"18f47428-da97-456a-8200-5f665c9f5d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ffd208e-bkvnb"
Nov 03 18:11:16.275987 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:16.275948    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18f47428-da97-456a-8200-5f665c9f5d46-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-5ffd208e-bkvnb\" (UID: \"18f47428-da97-456a-8200-5f665c9f5d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ffd208e-bkvnb"
Nov 03 18:11:16.276166 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:16.276131    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05817bd1-81be-4585-b1c2-0f3ca80eec9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-260dd0-5469bdccd7-82scv\" (UID: \"05817bd1-81be-4585-b1c2-0f3ca80eec9d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-260dd0-5469bdccd7-82scv"
Nov 03 18:11:16.276276 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:16.276196    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/232bbb8c-f8b8-42a7-a29c-f288fe7e99f9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-260dd0-76766ff95-8999v\" (UID: \"232bbb8c-f8b8-42a7-a29c-f288fe7e99f9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-260dd0-76766ff95-8999v"
Nov 03 18:11:16.278372 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:16.278349    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18f47428-da97-456a-8200-5f665c9f5d46-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-5ffd208e-bkvnb\" (UID: \"18f47428-da97-456a-8200-5f665c9f5d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ffd208e-bkvnb"
Nov 03 18:11:16.358223 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:16.358132    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bshlb\" (UniqueName: \"kubernetes.io/projected/232bbb8c-f8b8-42a7-a29c-f288fe7e99f9-kube-api-access-bshlb\") pod \"payload-processing-e2e-shared-b-w4-260dd0-76766ff95-8999v\" (UID: \"232bbb8c-f8b8-42a7-a29c-f288fe7e99f9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-260dd0-76766ff95-8999v"
Nov 03 18:11:16.358223 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:16.358165    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqxn4\" (UniqueName: \"kubernetes.io/projected/05817bd1-81be-4585-b1c2-0f3ca80eec9d-kube-api-access-tqxn4\") pod \"payload-pre-processing-e2e-shared-b-w4-260dd0-5469bdccd7-82scv\" (UID: \"05817bd1-81be-4585-b1c2-0f3ca80eec9d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-260dd0-5469bdccd7-82scv"
Nov 03 18:11:16.358947 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:16.358922    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltzwc\" (UniqueName: \"kubernetes.io/projected/18f47428-da97-456a-8200-5f665c9f5d46-kube-api-access-ltzwc\") pod \"maas-api-revoke-keys-e2e-derive-5ffd208e-bkvnb\" (UID: \"18f47428-da97-456a-8200-5f665c9f5d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ffd208e-bkvnb"
Nov 03 18:11:16.412368 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:16.412323    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ffd208e-bkvnb"
Nov 03 18:11:16.457012 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:16.456973    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-260dd0-76766ff95-8999v"
Nov 03 18:11:16.480126 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:16.480088    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-260dd0-5469bdccd7-82scv"
Nov 03 18:11:16.663909 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:16.663860    5315 patch_prober.go:29] interesting pod/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 03 18:11:16.663992 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:16.663923    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67" podUID="d7899300-736f-4a03-aef7-253d7a1b1e2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 03 18:11:16.894624 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:16.894579    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ffd208e-bkvnb"]
Nov 03 18:11:16.897376 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:11:16.897347    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18f47428_da97_456a_8200_5f665c9f5d46.slice/crio-645e8c704c94b08b96788ee589ad6ffdb7555556aa3ce973f4bb9915278f6512 WatchSource:0}: Error finding container 645e8c704c94b08b96788ee589ad6ffdb7555556aa3ce973f4bb9915278f6512: Status 404 returned error can't find the container with id 645e8c704c94b08b96788ee589ad6ffdb7555556aa3ce973f4bb9915278f6512
Nov 03 18:11:16.997658 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:16.997602    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-260dd0-5469bdccd7-82scv"]
Nov 03 18:11:17.001263 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:11:17.001235    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05817bd1_81be_4585_b1c2_0f3ca80eec9d.slice/crio-9e097ea73994def176bdbd91bd8cb02953c2a9f5cfa1bce789d5144c026db238 WatchSource:0}: Error finding container 9e097ea73994def176bdbd91bd8cb02953c2a9f5cfa1bce789d5144c026db238: Status 404 returned error can't find the container with id 9e097ea73994def176bdbd91bd8cb02953c2a9f5cfa1bce789d5144c026db238
Nov 03 18:11:17.001484 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:17.001456    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ffd208e-bkvnb" event={"ID":"18f47428-da97-456a-8200-5f665c9f5d46","Type":"ContainerStarted","Data":"645e8c704c94b08b96788ee589ad6ffdb7555556aa3ce973f4bb9915278f6512"}
Nov 03 18:11:17.007494 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:17.007469    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-260dd0-76766ff95-8999v"]
Nov 03 18:11:17.011020 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:11:17.010994    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232bbb8c_f8b8_42a7_a29c_f288fe7e99f9.slice/crio-572f2e9b439a53ebf257bc6305fdc553b4ed9c6e965c6db1b1ea18bdd7429984 WatchSource:0}: Error finding container 572f2e9b439a53ebf257bc6305fdc553b4ed9c6e965c6db1b1ea18bdd7429984: Status 404 returned error can't find the container with id 572f2e9b439a53ebf257bc6305fdc553b4ed9c6e965c6db1b1ea18bdd7429984
Nov 03 18:11:17.666310 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:17.666278    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67"
Nov 03 18:11:17.666859 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:17.666836    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67"
Nov 03 18:11:17.667456 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:17.667436    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67"
Nov 03 18:11:18.032583 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:18.032539    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-260dd0-5469bdccd7-82scv" event={"ID":"05817bd1-81be-4585-b1c2-0f3ca80eec9d","Type":"ContainerStarted","Data":"205edba7c548de87bc5f9f181b6a86b719ad557bfaeb93c540842081ae226b1d"}
Nov 03 18:11:18.033070 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:18.032626    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-260dd0-5469bdccd7-82scv"
Nov 03 18:11:18.033070 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:18.032652    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-260dd0-5469bdccd7-82scv" event={"ID":"05817bd1-81be-4585-b1c2-0f3ca80eec9d","Type":"ContainerStarted","Data":"9e097ea73994def176bdbd91bd8cb02953c2a9f5cfa1bce789d5144c026db238"}
Nov 03 18:11:18.034504 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:18.034469    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ffd208e-bkvnb" event={"ID":"18f47428-da97-456a-8200-5f665c9f5d46","Type":"ContainerStarted","Data":"8fd5709be236f0591519b6d365dbfb7c00f5a186674198579b95d2ab1d8bb468"}
Nov 03 18:11:18.036167 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:18.036142    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-260dd0-76766ff95-8999v" event={"ID":"232bbb8c-f8b8-42a7-a29c-f288fe7e99f9","Type":"ContainerStarted","Data":"af7a33ccf822429da41bda0717e7a0d2d35d422652f1c35271b2255e5581c33f"}
Nov 03 18:11:18.036290 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:18.036192    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-260dd0-76766ff95-8999v" event={"ID":"232bbb8c-f8b8-42a7-a29c-f288fe7e99f9","Type":"ContainerStarted","Data":"572f2e9b439a53ebf257bc6305fdc553b4ed9c6e965c6db1b1ea18bdd7429984"}
Nov 03 18:11:18.036417 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:18.036401    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-260dd0-76766ff95-8999v"
Nov 03 18:11:18.048897 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:18.048876    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6814726-rtxxz"]
Nov 03 18:11:18.053559 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:18.053536    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6814726-rtxxz"]
Nov 03 18:11:18.142093 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:18.142045    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-260dd0-5469bdccd7-82scv" podStartSLOduration=3.1420323310000002 podStartE2EDuration="3.142032331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:11:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:11:18.139638833 +0000 UTC m=+1517.155836899" watchObservedRunningTime="2026-11-03 18:11:18.142032331 +0000 UTC m=+1517.158230400"
Nov 03 18:11:18.250990 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:18.250935    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ffd208e-bkvnb" podStartSLOduration=4.250921412 podStartE2EDuration="4.250921412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:11:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:11:18.247048392 +0000 UTC m=+1517.263246466" watchObservedRunningTime="2026-11-03 18:11:18.250921412 +0000 UTC m=+1517.267119475"
Nov 03 18:11:18.417562 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:18.417435    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-260dd0-76766ff95-8999v" podStartSLOduration=3.417417273 podStartE2EDuration="3.417417273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:11:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:11:18.366066227 +0000 UTC m=+1517.382264299" watchObservedRunningTime="2026-11-03 18:11:18.417417273 +0000 UTC m=+1517.433615342"
Nov 03 18:11:19.046190 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:19.046154    5315 generic.go:356] "Generic (PLEG): container finished" podID="18f47428-da97-456a-8200-5f665c9f5d46" containerID="8fd5709be236f0591519b6d365dbfb7c00f5a186674198579b95d2ab1d8bb468" exitCode=0
Nov 03 18:11:19.046584 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:19.046247    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ffd208e-bkvnb" event={"ID":"18f47428-da97-456a-8200-5f665c9f5d46","Type":"ContainerDied","Data":"8fd5709be236f0591519b6d365dbfb7c00f5a186674198579b95d2ab1d8bb468"}
Nov 03 18:11:19.106444 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:19.106408    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3224261c-8cfa-4f55-90e6-cbbd099f8de7" path="/var/lib/kubelet/pods/3224261c-8cfa-4f55-90e6-cbbd099f8de7/volumes"
Nov 03 18:11:20.048002 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:20.047968    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daea96db-846wr"]
Nov 03 18:11:20.094708 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:20.094675    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daea96db-846wr"]
Nov 03 18:11:20.247516 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:20.247489    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ffd208e-bkvnb"
Nov 03 18:11:20.272490 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:20.272456    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18f47428-da97-456a-8200-5f665c9f5d46-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18f47428-da97-456a-8200-5f665c9f5d46-maas-api-xks-ca\") pod \"18f47428-da97-456a-8200-5f665c9f5d46\" (UID: \"18f47428-da97-456a-8200-5f665c9f5d46\") "
Nov 03 18:11:20.272642 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:20.272564    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18f47428-da97-456a-8200-5f665c9f5d46-kube-api-access-ltzwc\" (UniqueName: \"kubernetes.io/projected/18f47428-da97-456a-8200-5f665c9f5d46-kube-api-access-ltzwc\") pod \"18f47428-da97-456a-8200-5f665c9f5d46\" (UID: \"18f47428-da97-456a-8200-5f665c9f5d46\") "
Nov 03 18:11:20.272858 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:20.272830    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18f47428-da97-456a-8200-5f665c9f5d46-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18f47428-da97-456a-8200-5f665c9f5d46-maas-api-openshift-ca\") pod \"18f47428-da97-456a-8200-5f665c9f5d46\" (UID: \"18f47428-da97-456a-8200-5f665c9f5d46\") "
Nov 03 18:11:20.273199 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:20.273165    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18f47428-da97-456a-8200-5f665c9f5d46-maas-api-openshift-ca" pod "18f47428-da97-456a-8200-5f665c9f5d46" (UID: "18f47428-da97-456a-8200-5f665c9f5d46"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:11:20.273722 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:20.273700    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18f47428-da97-456a-8200-5f665c9f5d46-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:11:20.274656 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:20.274628    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18f47428-da97-456a-8200-5f665c9f5d46-maas-api-xks-ca" pod "18f47428-da97-456a-8200-5f665c9f5d46" (UID: "18f47428-da97-456a-8200-5f665c9f5d46"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:11:20.274788 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:20.274766    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18f47428-da97-456a-8200-5f665c9f5d46-kube-api-access-ltzwc" pod "18f47428-da97-456a-8200-5f665c9f5d46" (UID: "18f47428-da97-456a-8200-5f665c9f5d46"). InnerVolumeSpecName "kube-api-access-ltzwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:11:20.375712 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:20.375621    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18f47428-da97-456a-8200-5f665c9f5d46-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:11:20.375712 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:20.375659    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltzwc\" (UniqueName: \"kubernetes.io/projected/18f47428-da97-456a-8200-5f665c9f5d46-kube-api-access-ltzwc\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:11:21.065557 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:21.065523    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ffd208e-bkvnb"
Nov 03 18:11:21.065557 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:21.065540    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ffd208e-bkvnb" event={"ID":"18f47428-da97-456a-8200-5f665c9f5d46","Type":"ContainerDied","Data":"645e8c704c94b08b96788ee589ad6ffdb7555556aa3ce973f4bb9915278f6512"}
Nov 03 18:11:21.066139 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:21.065632    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="645e8c704c94b08b96788ee589ad6ffdb7555556aa3ce973f4bb9915278f6512"
Nov 03 18:11:21.106217 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:21.106186    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98" path="/var/lib/kubelet/pods/327ab1b6-a43b-4f94-ba1b-15f4cc3b4e98/volumes"
Nov 03 18:11:22.048136 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:22.048094    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-227429c2-kl7ck"]
Nov 03 18:11:22.094537 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:22.094505    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-227429c2-kl7ck"]
Nov 03 18:11:23.106515 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:23.106480    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0e395dc-0519-4588-aa3e-aa9067700361" path="/var/lib/kubelet/pods/f0e395dc-0519-4588-aa3e-aa9067700361/volumes"
Nov 03 18:11:24.580094 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:11:24.580053    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:11:26.432851 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:26.432816    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5ffd208e-5c84c4495f-nk9hq"]
Nov 03 18:11:26.544064 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:26.544029    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5ffd208e-64b984695-8jftk"]
Nov 03 18:11:27.936798 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:27.936755    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d"]
Nov 03 18:11:27.939123 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:27.939096    5315 cpu_manager.go:414] "containerMap: removing container" podUID="18f47428-da97-456a-8200-5f665c9f5d46" containerName="revoke-keys"
Nov 03 18:11:27.939318 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:27.939289    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18f47428-da97-456a-8200-5f665c9f5d46" containerName="revoke-keys"
Nov 03 18:11:27.939810 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:27.939780    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="18f47428-da97-456a-8200-5f665c9f5d46" containerName="revoke-keys"
Nov 03 18:11:27.950155 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:27.950125    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d"
Nov 03 18:11:28.001660 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:28.001605    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-7a7c76-openshift-default-dockercfg-6b4nn\""
Nov 03 18:11:28.027651 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:28.027585    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d"]
Nov 03 18:11:28.059054 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:28.057483    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-credential-socket\") pod \"e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d\" (UID: \"a91acd67-7bae-4d10-a5b0-4f1d4d81253f\") " pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d"
Nov 03 18:11:28.059445 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:28.059416    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvv58\" (UniqueName: \"kubernetes.io/projected/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-kube-api-access-kvv58\") pod \"e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d\" (UID: \"a91acd67-7bae-4d10-a5b0-4f1d4d81253f\") " pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d"
Nov 03 18:11:28.059664 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:28.059643    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-istio-token\") pod \"e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d\" (UID: \"a91acd67-7bae-4d10-a5b0-4f1d4d81253f\") " pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d"
Nov 03 18:11:28.059892 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:28.059874    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-istio-podinfo\") pod \"e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d\" (UID: \"a91acd67-7bae-4d10-a5b0-4f1d4d81253f\") " pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d"
Nov 03 18:11:28.060113 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:28.060094    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-istiod-ca-cert\") pod \"e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d\" (UID: \"a91acd67-7bae-4d10-a5b0-4f1d4d81253f\") " pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d"
Nov 03 18:11:28.060354 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:28.060336    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-workload-socket\") pod \"e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d\" (UID: \"a91acd67-7bae-4d10-a5b0-4f1d4d81253f\") " pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d"
Nov 03 18:11:28.060625 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:28.060584    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-workload-certs\") pod \"e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d\" (UID: \"a91acd67-7bae-4d10-a5b0-4f1d4d81253f\") " pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d"
Nov 03 18:11:28.060806 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:28.060789    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-istio-data\") pod \"e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d\" (UID: \"a91acd67-7bae-4d10-a5b0-4f1d4d81253f\") " pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d"
Nov 03 18:11:28.061042 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:28.061022    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-istio-envoy\") pod \"e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d\" (UID: \"a91acd67-7bae-4d10-a5b0-4f1d4d81253f\") " pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d"
Nov 03 18:11:28.238273 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:28.238154    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-workload-socket\") pod \"e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d\" (UID: \"a91acd67-7bae-4d10-a5b0-4f1d4d81253f\") " pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d"
Nov 03 18:11:28.239172 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:28.239135    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-workload-certs\") pod \"e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d\" (UID: \"a91acd67-7bae-4d10-a5b0-4f1d4d81253f\") " pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d"
Nov 03 18:11:28.239468 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:28.239448    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-istio-data\") pod \"e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d\" (UID: \"a91acd67-7bae-4d10-a5b0-4f1d4d81253f\") " pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d"
Nov 03 18:11:28.239751 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:28.239732    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-istio-envoy\") pod \"e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d\" (UID: \"a91acd67-7bae-4d10-a5b0-4f1d4d81253f\") " pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d"
Nov 03 18:11:28.240157 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:28.240133    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-credential-socket\") pod \"e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d\" (UID: \"a91acd67-7bae-4d10-a5b0-4f1d4d81253f\") " pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d"
Nov 03 18:11:28.257496 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:28.252523    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-istio-data\") pod \"e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d\" (UID: \"a91acd67-7bae-4d10-a5b0-4f1d4d81253f\") " pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d"
Nov 03 18:11:28.261686 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:28.254323    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-credential-socket\") pod \"e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d\" (UID: \"a91acd67-7bae-4d10-a5b0-4f1d4d81253f\") " pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d"
Nov 03 18:11:28.261881 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:28.254701    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvv58\" (UniqueName: \"kubernetes.io/projected/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-kube-api-access-kvv58\") pod \"e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d\" (UID: \"a91acd67-7bae-4d10-a5b0-4f1d4d81253f\") " pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d"
Nov 03 18:11:28.262026 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:28.262001    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-istio-token\") pod \"e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d\" (UID: \"a91acd67-7bae-4d10-a5b0-4f1d4d81253f\") " pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d"
Nov 03 18:11:28.269019 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:28.268985    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-istio-podinfo\") pod \"e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d\" (UID: \"a91acd67-7bae-4d10-a5b0-4f1d4d81253f\") " pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d"
Nov 03 18:11:28.276662 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:28.269295    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-istiod-ca-cert\") pod \"e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d\" (UID: \"a91acd67-7bae-4d10-a5b0-4f1d4d81253f\") " pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d"
Nov 03 18:11:28.277130 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:28.272874    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-workload-certs\") pod \"e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d\" (UID: \"a91acd67-7bae-4d10-a5b0-4f1d4d81253f\") " pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d"
Nov 03 18:11:28.301755 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:28.238975    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-workload-socket\") pod \"e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d\" (UID: \"a91acd67-7bae-4d10-a5b0-4f1d4d81253f\") " pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d"
Nov 03 18:11:28.303264 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:28.273897    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-istiod-ca-cert\") pod \"e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d\" (UID: \"a91acd67-7bae-4d10-a5b0-4f1d4d81253f\") " pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d"
Nov 03 18:11:28.303264 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:28.278639    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-istio-podinfo\") pod \"e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d\" (UID: \"a91acd67-7bae-4d10-a5b0-4f1d4d81253f\") " pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d"
Nov 03 18:11:28.306732 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:28.305939    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-istio-envoy\") pod \"e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d\" (UID: \"a91acd67-7bae-4d10-a5b0-4f1d4d81253f\") " pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d"
Nov 03 18:11:28.330644 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:28.328504    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvv58\" (UniqueName: \"kubernetes.io/projected/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-kube-api-access-kvv58\") pod \"e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d\" (UID: \"a91acd67-7bae-4d10-a5b0-4f1d4d81253f\") " pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d"
Nov 03 18:11:28.338629 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:28.331268    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-istio-token\") pod \"e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d\" (UID: \"a91acd67-7bae-4d10-a5b0-4f1d4d81253f\") " pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d"
Nov 03 18:11:28.602214 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:28.601859    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d"
Nov 03 18:11:29.049488 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:29.049447    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-260dd0-5469bdccd7-82scv"
Nov 03 18:11:29.049936 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:29.049534    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-260dd0-76766ff95-8999v"
Nov 03 18:11:29.054488 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:29.054459    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80bee53a-zmp7n"]
Nov 03 18:11:29.059706 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:29.059677    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80bee53a-zmp7n"]
Nov 03 18:11:29.106742 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:29.106663    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2458d282-5d5f-4967-8536-85ba89c9eec3" path="/var/lib/kubelet/pods/2458d282-5d5f-4967-8536-85ba89c9eec3/volumes"
Nov 03 18:11:29.231803 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:29.231771    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d"]
Nov 03 18:11:29.235997 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:11:29.235968    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda91acd67_7bae_4d10_a5b0_4f1d4d81253f.slice/crio-e59092dc0fc453af64e3e1f625b3939226187ebc7cda16b952f17df1a7a6253f WatchSource:0}: Error finding container e59092dc0fc453af64e3e1f625b3939226187ebc7cda16b952f17df1a7a6253f: Status 404 returned error can't find the container with id e59092dc0fc453af64e3e1f625b3939226187ebc7cda16b952f17df1a7a6253f
Nov 03 18:11:29.238770 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:29.238735    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:11:29.238841 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:29.238804    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:11:29.238841 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:29.238832    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:11:29.336850 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:29.336812    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d" event={"ID":"a91acd67-7bae-4d10-a5b0-4f1d4d81253f","Type":"ContainerStarted","Data":"e59092dc0fc453af64e3e1f625b3939226187ebc7cda16b952f17df1a7a6253f"}
Nov 03 18:11:30.052286 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:30.052251    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4298e49-xjwqp"]
Nov 03 18:11:30.057086 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:30.057057    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4298e49-xjwqp"]
Nov 03 18:11:30.347202 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:30.347109    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d" event={"ID":"a91acd67-7bae-4d10-a5b0-4f1d4d81253f","Type":"ContainerStarted","Data":"5a807df74fdb3586c786c96521db6b16a152470bd59c19a4be212fcd36135eaa"}
Nov 03 18:11:30.463211 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:30.463149    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d" podStartSLOduration=3.463132758 podStartE2EDuration="3.463132758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:11:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:11:30.459679897 +0000 UTC m=+1529.475877967" watchObservedRunningTime="2026-11-03 18:11:30.463132758 +0000 UTC m=+1529.479330826"
Nov 03 18:11:30.556820 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:30.555111    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77185c03-2nn69"]
Nov 03 18:11:30.562149 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:30.562116    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77185c03-2nn69"
Nov 03 18:11:30.614079 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:30.613989    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:11:30.616049 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:30.615998    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d"
Nov 03 18:11:30.616704 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:30.616682    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:11:30.617038 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:30.617023    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:11:30.617310 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:30.617204    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:11:30.655533 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:30.654231    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77185c03-2nn69"]
Nov 03 18:11:30.686632 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:30.680777    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d"
Nov 03 18:11:30.750032 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:30.747259    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69511ea9-d925-43c9-a861-5cf16ee6b490-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77185c03-2nn69\" (UID: \"69511ea9-d925-43c9-a861-5cf16ee6b490\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77185c03-2nn69"
Nov 03 18:11:30.750032 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:30.747412    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4lk6\" (UniqueName: \"kubernetes.io/projected/69511ea9-d925-43c9-a861-5cf16ee6b490-kube-api-access-z4lk6\") pod \"maas-api-revoke-sub-77185c03-2nn69\" (UID: \"69511ea9-d925-43c9-a861-5cf16ee6b490\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77185c03-2nn69"
Nov 03 18:11:30.750032 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:30.747640    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69511ea9-d925-43c9-a861-5cf16ee6b490-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77185c03-2nn69\" (UID: \"69511ea9-d925-43c9-a861-5cf16ee6b490\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77185c03-2nn69"
Nov 03 18:11:30.849668 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:30.849629    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69511ea9-d925-43c9-a861-5cf16ee6b490-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77185c03-2nn69\" (UID: \"69511ea9-d925-43c9-a861-5cf16ee6b490\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77185c03-2nn69"
Nov 03 18:11:30.849864 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:30.849792    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69511ea9-d925-43c9-a861-5cf16ee6b490-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77185c03-2nn69\" (UID: \"69511ea9-d925-43c9-a861-5cf16ee6b490\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77185c03-2nn69"
Nov 03 18:11:30.849959 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:30.849927    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4lk6\" (UniqueName: \"kubernetes.io/projected/69511ea9-d925-43c9-a861-5cf16ee6b490-kube-api-access-z4lk6\") pod \"maas-api-revoke-sub-77185c03-2nn69\" (UID: \"69511ea9-d925-43c9-a861-5cf16ee6b490\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77185c03-2nn69"
Nov 03 18:11:30.850716 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:30.850684    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69511ea9-d925-43c9-a861-5cf16ee6b490-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77185c03-2nn69\" (UID: \"69511ea9-d925-43c9-a861-5cf16ee6b490\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77185c03-2nn69"
Nov 03 18:11:30.852751 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:30.852725    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69511ea9-d925-43c9-a861-5cf16ee6b490-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77185c03-2nn69\" (UID: \"69511ea9-d925-43c9-a861-5cf16ee6b490\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77185c03-2nn69"
Nov 03 18:11:30.907604 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:30.907535    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4lk6\" (UniqueName: \"kubernetes.io/projected/69511ea9-d925-43c9-a861-5cf16ee6b490-kube-api-access-z4lk6\") pod \"maas-api-revoke-sub-77185c03-2nn69\" (UID: \"69511ea9-d925-43c9-a861-5cf16ee6b490\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77185c03-2nn69"
Nov 03 18:11:31.109877 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:31.109843    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="518883f2-c175-4ad7-a4d0-15dfa5520315" path="/var/lib/kubelet/pods/518883f2-c175-4ad7-a4d0-15dfa5520315/volumes"
Nov 03 18:11:31.196072 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:31.195975    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77185c03-2nn69"
Nov 03 18:11:31.358004 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:31.357970    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d"
Nov 03 18:11:31.359150 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:31.359123    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d"
Nov 03 18:11:31.433318 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:31.433269    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-5ffd208e-5c84c4495f-nk9hq" podUID="d9b0b852-1183-4ae0-87a0-6832aa16c3d9" containerName="payload-processing" containerID="cri-o://6591c34c4d66ebcd678cbeca98781d21919cd7fdc2c38981ca3b11255a913f94" gracePeriod=25
Nov 03 18:11:31.544650 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:31.544546    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-5ffd208e-64b984695-8jftk" podUID="79925de9-7aba-4361-bc74-40ecc8f6bee2" containerName="payload-pre-processing" containerID="cri-o://4ae8d7f08e897610285ec25ee70df1b8580205cb0ab3dbb97d8d015da666e0a9" gracePeriod=25
Nov 03 18:11:31.620894 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:31.620597    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77185c03-2nn69"]
Nov 03 18:11:31.628252 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:11:31.628189    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69511ea9_d925_43c9_a861_5cf16ee6b490.slice/crio-cc22594f0cb2fc5c3fc65d00d2fb93bccb34149ebd0a360913e7db428a79e859 WatchSource:0}: Error finding container cc22594f0cb2fc5c3fc65d00d2fb93bccb34149ebd0a360913e7db428a79e859: Status 404 returned error can't find the container with id cc22594f0cb2fc5c3fc65d00d2fb93bccb34149ebd0a360913e7db428a79e859
Nov 03 18:11:31.726946 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:31.726914    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-5ffd208e-5c84c4495f-nk9hq"
Nov 03 18:11:31.768628 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:31.768533    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9b0b852-1183-4ae0-87a0-6832aa16c3d9-kube-api-access-lmbtr\" (UniqueName: \"kubernetes.io/projected/d9b0b852-1183-4ae0-87a0-6832aa16c3d9-kube-api-access-lmbtr\") pod \"d9b0b852-1183-4ae0-87a0-6832aa16c3d9\" (UID: \"d9b0b852-1183-4ae0-87a0-6832aa16c3d9\") "
Nov 03 18:11:31.768803 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:31.768677    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9b0b852-1183-4ae0-87a0-6832aa16c3d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9b0b852-1183-4ae0-87a0-6832aa16c3d9-plugins-config-volume\") pod \"d9b0b852-1183-4ae0-87a0-6832aa16c3d9\" (UID: \"d9b0b852-1183-4ae0-87a0-6832aa16c3d9\") "
Nov 03 18:11:31.769282 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:31.769254    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9b0b852-1183-4ae0-87a0-6832aa16c3d9-plugins-config-volume" pod "d9b0b852-1183-4ae0-87a0-6832aa16c3d9" (UID: "d9b0b852-1183-4ae0-87a0-6832aa16c3d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:11:31.769500 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:31.769479    5315 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9b0b852-1183-4ae0-87a0-6832aa16c3d9-plugins-config-volume\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:11:31.825409 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:31.825362    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9b0b852-1183-4ae0-87a0-6832aa16c3d9-kube-api-access-lmbtr" pod "d9b0b852-1183-4ae0-87a0-6832aa16c3d9" (UID: "d9b0b852-1183-4ae0-87a0-6832aa16c3d9"). InnerVolumeSpecName "kube-api-access-lmbtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:11:31.871818 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:31.871784    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmbtr\" (UniqueName: \"kubernetes.io/projected/d9b0b852-1183-4ae0-87a0-6832aa16c3d9-kube-api-access-lmbtr\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:11:31.900625 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:31.900580    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5ffd208e-64b984695-8jftk"
Nov 03 18:11:31.973065 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:31.972982    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79925de9-7aba-4361-bc74-40ecc8f6bee2-kube-api-access-wh2rc\" (UniqueName: \"kubernetes.io/projected/79925de9-7aba-4361-bc74-40ecc8f6bee2-kube-api-access-wh2rc\") pod \"79925de9-7aba-4361-bc74-40ecc8f6bee2\" (UID: \"79925de9-7aba-4361-bc74-40ecc8f6bee2\") "
Nov 03 18:11:31.973210 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:31.973090    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79925de9-7aba-4361-bc74-40ecc8f6bee2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79925de9-7aba-4361-bc74-40ecc8f6bee2-plugins-config-volume\") pod \"79925de9-7aba-4361-bc74-40ecc8f6bee2\" (UID: \"79925de9-7aba-4361-bc74-40ecc8f6bee2\") "
Nov 03 18:11:31.973435 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:31.973406    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79925de9-7aba-4361-bc74-40ecc8f6bee2-plugins-config-volume" pod "79925de9-7aba-4361-bc74-40ecc8f6bee2" (UID: "79925de9-7aba-4361-bc74-40ecc8f6bee2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:11:31.973924 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:31.973902    5315 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79925de9-7aba-4361-bc74-40ecc8f6bee2-plugins-config-volume\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:11:32.024383 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:32.024336    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79925de9-7aba-4361-bc74-40ecc8f6bee2-kube-api-access-wh2rc" pod "79925de9-7aba-4361-bc74-40ecc8f6bee2" (UID: "79925de9-7aba-4361-bc74-40ecc8f6bee2"). InnerVolumeSpecName "kube-api-access-wh2rc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:11:32.075681 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:32.075649    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh2rc\" (UniqueName: \"kubernetes.io/projected/79925de9-7aba-4361-bc74-40ecc8f6bee2-kube-api-access-wh2rc\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:11:32.369078 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:32.369041    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77185c03-2nn69" event={"ID":"69511ea9-d925-43c9-a861-5cf16ee6b490","Type":"ContainerStarted","Data":"373421a227e145d568570e337731ca87dcd3a430773fb217f9ae78dc2982df23"}
Nov 03 18:11:32.369569 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:32.369100    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77185c03-2nn69" event={"ID":"69511ea9-d925-43c9-a861-5cf16ee6b490","Type":"ContainerStarted","Data":"cc22594f0cb2fc5c3fc65d00d2fb93bccb34149ebd0a360913e7db428a79e859"}
Nov 03 18:11:32.370594 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:32.370564    5315 generic.go:356] "Generic (PLEG): container finished" podID="d9b0b852-1183-4ae0-87a0-6832aa16c3d9" containerID="6591c34c4d66ebcd678cbeca98781d21919cd7fdc2c38981ca3b11255a913f94" exitCode=0
Nov 03 18:11:32.370718 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:32.370620    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-5ffd208e-5c84c4495f-nk9hq"
Nov 03 18:11:32.370718 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:32.370666    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-5ffd208e-5c84c4495f-nk9hq" event={"ID":"d9b0b852-1183-4ae0-87a0-6832aa16c3d9","Type":"ContainerDied","Data":"6591c34c4d66ebcd678cbeca98781d21919cd7fdc2c38981ca3b11255a913f94"}
Nov 03 18:11:32.370718 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:32.370716    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-5ffd208e-5c84c4495f-nk9hq" event={"ID":"d9b0b852-1183-4ae0-87a0-6832aa16c3d9","Type":"ContainerDied","Data":"2f8fa6e0daafc3e04ecd37970a19662bebaa3a3e1879280e7bc2932cc0abfe7d"}
Nov 03 18:11:32.370882 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:32.370744    5315 scope.go:122] "RemoveContainer" containerID="6591c34c4d66ebcd678cbeca98781d21919cd7fdc2c38981ca3b11255a913f94"
Nov 03 18:11:32.372159 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:32.372138    5315 generic.go:356] "Generic (PLEG): container finished" podID="79925de9-7aba-4361-bc74-40ecc8f6bee2" containerID="4ae8d7f08e897610285ec25ee70df1b8580205cb0ab3dbb97d8d015da666e0a9" exitCode=0
Nov 03 18:11:32.372222 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:32.372200    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5ffd208e-64b984695-8jftk"
Nov 03 18:11:32.372273 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:32.372218    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5ffd208e-64b984695-8jftk" event={"ID":"79925de9-7aba-4361-bc74-40ecc8f6bee2","Type":"ContainerDied","Data":"4ae8d7f08e897610285ec25ee70df1b8580205cb0ab3dbb97d8d015da666e0a9"}
Nov 03 18:11:32.372325 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:32.372277    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5ffd208e-64b984695-8jftk" event={"ID":"79925de9-7aba-4361-bc74-40ecc8f6bee2","Type":"ContainerDied","Data":"28f6547803261d1bd8c3932755eb801c0836a3b7b2f681643ca7236e2bc796f8"}
Nov 03 18:11:32.383686 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:32.383628    5315 scope.go:122] "RemoveContainer" containerID="6591c34c4d66ebcd678cbeca98781d21919cd7fdc2c38981ca3b11255a913f94"
Nov 03 18:11:32.384439 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:11:32.384417    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6591c34c4d66ebcd678cbeca98781d21919cd7fdc2c38981ca3b11255a913f94\": container with ID starting with 6591c34c4d66ebcd678cbeca98781d21919cd7fdc2c38981ca3b11255a913f94 not found: ID does not exist" containerID="6591c34c4d66ebcd678cbeca98781d21919cd7fdc2c38981ca3b11255a913f94"
Nov 03 18:11:32.384527 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:32.384444    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6591c34c4d66ebcd678cbeca98781d21919cd7fdc2c38981ca3b11255a913f94"} err="failed to get container status \"6591c34c4d66ebcd678cbeca98781d21919cd7fdc2c38981ca3b11255a913f94\": rpc error: code = NotFound desc = could not find container \"6591c34c4d66ebcd678cbeca98781d21919cd7fdc2c38981ca3b11255a913f94\": container with ID starting with 6591c34c4d66ebcd678cbeca98781d21919cd7fdc2c38981ca3b11255a913f94 not found: ID does not exist"
Nov 03 18:11:32.384527 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:32.384462    5315 scope.go:122] "RemoveContainer" containerID="4ae8d7f08e897610285ec25ee70df1b8580205cb0ab3dbb97d8d015da666e0a9"
Nov 03 18:11:32.395465 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:32.395445    5315 scope.go:122] "RemoveContainer" containerID="4ae8d7f08e897610285ec25ee70df1b8580205cb0ab3dbb97d8d015da666e0a9"
Nov 03 18:11:32.395803 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:11:32.395780    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ae8d7f08e897610285ec25ee70df1b8580205cb0ab3dbb97d8d015da666e0a9\": container with ID starting with 4ae8d7f08e897610285ec25ee70df1b8580205cb0ab3dbb97d8d015da666e0a9 not found: ID does not exist" containerID="4ae8d7f08e897610285ec25ee70df1b8580205cb0ab3dbb97d8d015da666e0a9"
Nov 03 18:11:32.395908 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:32.395809    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ae8d7f08e897610285ec25ee70df1b8580205cb0ab3dbb97d8d015da666e0a9"} err="failed to get container status \"4ae8d7f08e897610285ec25ee70df1b8580205cb0ab3dbb97d8d015da666e0a9\": rpc error: code = NotFound desc = could not find container \"4ae8d7f08e897610285ec25ee70df1b8580205cb0ab3dbb97d8d015da666e0a9\": container with ID starting with 4ae8d7f08e897610285ec25ee70df1b8580205cb0ab3dbb97d8d015da666e0a9 not found: ID does not exist"
Nov 03 18:11:32.478264 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:32.478212    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77185c03-2nn69" podStartSLOduration=2.478198964 podStartE2EDuration="2.478198964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:11:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:11:32.476492023 +0000 UTC m=+1531.492690091" watchObservedRunningTime="2026-11-03 18:11:32.478198964 +0000 UTC m=+1531.494397033"
Nov 03 18:11:32.583809 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:32.583772    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5ffd208e-64b984695-8jftk"]
Nov 03 18:11:32.591522 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:32.590228    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5ffd208e-64b984695-8jftk"]
Nov 03 18:11:32.692745 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:32.692705    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5ffd208e-5c84c4495f-nk9hq"]
Nov 03 18:11:32.697901 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:32.697861    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5ffd208e-5c84c4495f-nk9hq"]
Nov 03 18:11:33.106822 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:33.106788    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79925de9-7aba-4361-bc74-40ecc8f6bee2" path="/var/lib/kubelet/pods/79925de9-7aba-4361-bc74-40ecc8f6bee2/volumes"
Nov 03 18:11:33.107298 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:33.107283    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9b0b852-1183-4ae0-87a0-6832aa16c3d9" path="/var/lib/kubelet/pods/d9b0b852-1183-4ae0-87a0-6832aa16c3d9/volumes"
Nov 03 18:11:33.383140 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:33.383046    5315 generic.go:356] "Generic (PLEG): container finished" podID="69511ea9-d925-43c9-a861-5cf16ee6b490" containerID="373421a227e145d568570e337731ca87dcd3a430773fb217f9ae78dc2982df23" exitCode=0
Nov 03 18:11:33.383555 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:33.383140    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77185c03-2nn69" event={"ID":"69511ea9-d925-43c9-a861-5cf16ee6b490","Type":"ContainerDied","Data":"373421a227e145d568570e337731ca87dcd3a430773fb217f9ae78dc2982df23"}
Nov 03 18:11:34.582075 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:34.582041    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77185c03-2nn69"
Nov 03 18:11:34.651357 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:34.651269    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69511ea9-d925-43c9-a861-5cf16ee6b490-kube-api-access-z4lk6\" (UniqueName: \"kubernetes.io/projected/69511ea9-d925-43c9-a861-5cf16ee6b490-kube-api-access-z4lk6\") pod \"69511ea9-d925-43c9-a861-5cf16ee6b490\" (UID: \"69511ea9-d925-43c9-a861-5cf16ee6b490\") "
Nov 03 18:11:34.651497 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:34.651458    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69511ea9-d925-43c9-a861-5cf16ee6b490-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69511ea9-d925-43c9-a861-5cf16ee6b490-maas-api-xks-ca\") pod \"69511ea9-d925-43c9-a861-5cf16ee6b490\" (UID: \"69511ea9-d925-43c9-a861-5cf16ee6b490\") "
Nov 03 18:11:34.651569 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:34.651551    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69511ea9-d925-43c9-a861-5cf16ee6b490-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69511ea9-d925-43c9-a861-5cf16ee6b490-maas-api-openshift-ca\") pod \"69511ea9-d925-43c9-a861-5cf16ee6b490\" (UID: \"69511ea9-d925-43c9-a861-5cf16ee6b490\") "
Nov 03 18:11:34.651910 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:34.651887    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69511ea9-d925-43c9-a861-5cf16ee6b490-maas-api-openshift-ca" pod "69511ea9-d925-43c9-a861-5cf16ee6b490" (UID: "69511ea9-d925-43c9-a861-5cf16ee6b490"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:11:34.652264 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:34.652242    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69511ea9-d925-43c9-a861-5cf16ee6b490-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:11:34.653557 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:34.653531    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69511ea9-d925-43c9-a861-5cf16ee6b490-maas-api-xks-ca" pod "69511ea9-d925-43c9-a861-5cf16ee6b490" (UID: "69511ea9-d925-43c9-a861-5cf16ee6b490"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:11:34.653653 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:34.653563    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69511ea9-d925-43c9-a861-5cf16ee6b490-kube-api-access-z4lk6" pod "69511ea9-d925-43c9-a861-5cf16ee6b490" (UID: "69511ea9-d925-43c9-a861-5cf16ee6b490"). InnerVolumeSpecName "kube-api-access-z4lk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:11:34.754066 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:34.754022    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4lk6\" (UniqueName: \"kubernetes.io/projected/69511ea9-d925-43c9-a861-5cf16ee6b490-kube-api-access-z4lk6\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:11:34.754066 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:34.754056    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69511ea9-d925-43c9-a861-5cf16ee6b490-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:11:35.406583 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:35.406495    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77185c03-2nn69"
Nov 03 18:11:35.406583 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:35.406526    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77185c03-2nn69" event={"ID":"69511ea9-d925-43c9-a861-5cf16ee6b490","Type":"ContainerDied","Data":"cc22594f0cb2fc5c3fc65d00d2fb93bccb34149ebd0a360913e7db428a79e859"}
Nov 03 18:11:35.406583 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:35.406575    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc22594f0cb2fc5c3fc65d00d2fb93bccb34149ebd0a360913e7db428a79e859"
Nov 03 18:11:38.055910 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:38.055871    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-683d0211-tgvbp"]
Nov 03 18:11:38.063389 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:38.063357    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-683d0211-tgvbp"]
Nov 03 18:11:39.105988 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:39.105940    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e44f53a6-9661-4774-b376-61594363d002" path="/var/lib/kubelet/pods/e44f53a6-9661-4774-b376-61594363d002/volumes"
Nov 03 18:11:44.055638 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:44.055520    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca9298d0-wkhpc"]
Nov 03 18:11:44.098970 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:44.098936    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca9298d0-wkhpc"]
Nov 03 18:11:45.108377 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:45.108332    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80bf3ec8-d27c-4ddd-a862-f5c405b11349" path="/var/lib/kubelet/pods/80bf3ec8-d27c-4ddd-a862-f5c405b11349/volumes"
Nov 03 18:11:52.048376 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:52.048330    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaaeec56-qxbj2"]
Nov 03 18:11:52.094155 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:52.094115    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaaeec56-qxbj2"]
Nov 03 18:11:52.836565 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:52.836527    5315 generic.go:356] "Generic (PLEG): container finished" podID="f85c57a2-8cb8-4403-ba55-1de5bf63a921" containerID="095a31f3667a5bef8c64a5ae510403143a89669654e53377a42100159f0dd681" exitCode=0
Nov 03 18:11:52.836848 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:52.836603    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w" event={"ID":"f85c57a2-8cb8-4403-ba55-1de5bf63a921","Type":"ContainerDied","Data":"095a31f3667a5bef8c64a5ae510403143a89669654e53377a42100159f0dd681"}
Nov 03 18:11:52.837198 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:52.837179    5315 scope.go:122] "RemoveContainer" containerID="095a31f3667a5bef8c64a5ae510403143a89669654e53377a42100159f0dd681"
Nov 03 18:11:52.839003 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:52.838971    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:11:52.839125 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:52.839056    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:11:52.839125 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:52.839106    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:11:53.156987 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:53.154924    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef" path="/var/lib/kubelet/pods/1fb39614-be67-47c2-a0a6-dd8d1bd6b2ef/volumes"
Nov 03 18:11:53.664869 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:53.664835    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w"
Nov 03 18:11:53.849030 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:53.848990    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w" event={"ID":"f85c57a2-8cb8-4403-ba55-1de5bf63a921","Type":"ContainerStarted","Data":"174a67c0fd81c58870ea0e9570ac58ddb00de44b552e94daab9ad7c2c61cf31a"}
Nov 03 18:11:54.053349 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:54.053300    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c9ad588-jp7t8"]
Nov 03 18:11:54.057982 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:54.057942    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c9ad588-jp7t8"]
Nov 03 18:11:54.581599 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:11:54.580595    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:11:54.648083 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:54.648045    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w"
Nov 03 18:11:54.649147 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:54.649112    5315 patch_prober.go:29] interesting pod/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 03 18:11:54.649306 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:54.649180    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w" podUID="f85c57a2-8cb8-4403-ba55-1de5bf63a921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 03 18:11:55.053849 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:55.053811    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3920dd9f-vrgft"]
Nov 03 18:11:55.107641 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:55.107565    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="051dcdb7-a6c4-4167-b814-8b16bb04fb4e" path="/var/lib/kubelet/pods/051dcdb7-a6c4-4167-b814-8b16bb04fb4e/volumes"
Nov 03 18:11:55.108030 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:55.108009    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3920dd9f-vrgft"]
Nov 03 18:11:55.649189 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:55.649148    5315 patch_prober.go:29] interesting pod/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 03 18:11:55.649746 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:55.649222    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w" podUID="f85c57a2-8cb8-4403-ba55-1de5bf63a921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 03 18:11:56.643134 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:56.643081    5315 patch_prober.go:29] interesting pod/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 03 18:11:56.643363 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:56.643159    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w" podUID="f85c57a2-8cb8-4403-ba55-1de5bf63a921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 03 18:11:57.107960 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:57.107926    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f" path="/var/lib/kubelet/pods/5cf954dc-53ea-4012-9ba6-fa0a5d85cd5f/volumes"
Nov 03 18:11:57.645053 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:57.645014    5315 patch_prober.go:29] interesting pod/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 03 18:11:57.645350 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:57.645316    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w" podUID="f85c57a2-8cb8-4403-ba55-1de5bf63a921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 03 18:11:58.644635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:58.638061    5315 patch_prober.go:29] interesting pod/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 03 18:11:58.644635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:58.638130    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w" podUID="f85c57a2-8cb8-4403-ba55-1de5bf63a921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 03 18:11:59.054600 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:59.054567    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99aae353-5k9vd"]
Nov 03 18:11:59.108566 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:59.108536    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99aae353-5k9vd"]
Nov 03 18:11:59.641922 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:59.641891    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w"
Nov 03 18:11:59.642390 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:59.642376    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w"
Nov 03 18:11:59.643137 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:11:59.643117    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w"
Nov 03 18:12:01.108900 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:01.108868    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55533f46-6666-4424-9d95-9187445ade85" path="/var/lib/kubelet/pods/55533f46-6666-4424-9d95-9187445ade85/volumes"
Nov 03 18:12:02.686445 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:02.686409    5315 scope.go:122] "RemoveContainer" containerID="3729e6fee38768f88fdd8d6bd886c8e4b1e061342613af9db14987a6425f3349"
Nov 03 18:12:02.707699 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:02.707669    5315 scope.go:122] "RemoveContainer" containerID="8c99bf09bc4b707293977c3d502136baadf7401621aa4c08586d11ee48303fc8"
Nov 03 18:12:02.721095 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:02.721073    5315 scope.go:122] "RemoveContainer" containerID="e222ef1c0e6116539c82ce902e9ace8e8e2c830e3ebd1f4e0dba5354529203d0"
Nov 03 18:12:02.758299 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:02.758270    5315 scope.go:122] "RemoveContainer" containerID="8a01bbaae0ee125f7fbd5adee0a7922fc4bf93d9d7a4a0462e8742f2898cb9ad"
Nov 03 18:12:02.776270 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:02.776237    5315 scope.go:122] "RemoveContainer" containerID="ee9d5c2dbdc7922a7f44703613015c2efa1357033170842c6a05c431280375da"
Nov 03 18:12:02.792357 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:02.792328    5315 scope.go:122] "RemoveContainer" containerID="e9094333e388d4000ebec138eb6815cc2c64f4b06b382afd9596eb9a2da39d79"
Nov 03 18:12:02.804641 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:02.804590    5315 scope.go:122] "RemoveContainer" containerID="1df464538912ad4267a02a083c6ea97c796071a5ccfb40a5ab938499fa3b799c"
Nov 03 18:12:02.815513 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:02.815463    5315 scope.go:122] "RemoveContainer" containerID="c5aca8edb998fc3a0c1a965b45ceb7cd9b8c0dee998fece6dff7b8cdd3a7e80f"
Nov 03 18:12:02.863697 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:02.863669    5315 scope.go:122] "RemoveContainer" containerID="0b714255af6ea54188c66fa9f4c2996c5423a28e2c49c30b2e28aff503f5d57e"
Nov 03 18:12:02.884764 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:02.884738    5315 scope.go:122] "RemoveContainer" containerID="306a5dc9f82a9f761f7d1af179ba9b920a5e34a83fbb2d2f7addc55c458651dc"
Nov 03 18:12:02.897454 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:02.897429    5315 scope.go:122] "RemoveContainer" containerID="3197f6e1c35cf3b2bd620b9cda58e8647a0eeab462449216feeba603c2379491"
Nov 03 18:12:02.908042 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:02.908022    5315 scope.go:122] "RemoveContainer" containerID="58f9d2a1f194916214c49301560e17d6ba2d32690e581b02b2b727bf8d19ac2a"
Nov 03 18:12:02.918724 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:02.918700    5315 scope.go:122] "RemoveContainer" containerID="a7eded8a39bf6654e1279aff2ed9e2edf2843a17f797958b84e640cbef9b69d9"
Nov 03 18:12:03.372680 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:03.364000    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ffd208e-bkvnb"]
Nov 03 18:12:03.405687 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:03.383285    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ffd208e-bkvnb"]
Nov 03 18:12:03.405687 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:03.383358    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c"]
Nov 03 18:12:03.405687 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:03.383595    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c" podUID="38acc846-2251-4000-ae98-e5a2492da6a6" containerName="istio-proxy" containerID="cri-o://3445112f581194bfc56d5fb23a22b94eff030673ff239c6b2118b93cdb1d9863" gracePeriod=30
Nov 03 18:12:04.792140 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:04.792104    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r"]
Nov 03 18:12:04.792962 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:04.792929    5315 cpu_manager.go:414] "containerMap: removing container" podUID="79925de9-7aba-4361-bc74-40ecc8f6bee2" containerName="payload-pre-processing"
Nov 03 18:12:04.792962 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:04.792965    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79925de9-7aba-4361-bc74-40ecc8f6bee2" containerName="payload-pre-processing"
Nov 03 18:12:04.793067 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:04.792981    5315 cpu_manager.go:414] "containerMap: removing container" podUID="69511ea9-d925-43c9-a861-5cf16ee6b490" containerName="revoke-keys"
Nov 03 18:12:04.793067 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:04.792988    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69511ea9-d925-43c9-a861-5cf16ee6b490" containerName="revoke-keys"
Nov 03 18:12:04.793067 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:04.793010    5315 cpu_manager.go:414] "containerMap: removing container" podUID="d9b0b852-1183-4ae0-87a0-6832aa16c3d9" containerName="payload-processing"
Nov 03 18:12:04.793067 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:04.793018    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9b0b852-1183-4ae0-87a0-6832aa16c3d9" containerName="payload-processing"
Nov 03 18:12:04.793261 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:04.793245    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="79925de9-7aba-4361-bc74-40ecc8f6bee2" containerName="payload-pre-processing"
Nov 03 18:12:04.793296 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:04.793263    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="69511ea9-d925-43c9-a861-5cf16ee6b490" containerName="revoke-keys"
Nov 03 18:12:04.793296 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:04.793274    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9b0b852-1183-4ae0-87a0-6832aa16c3d9" containerName="payload-processing"
Nov 03 18:12:04.799173 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:04.799151    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r"
Nov 03 18:12:04.857634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:04.848361    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-80aca1-openshift-default-dockercfg-wsndj\""
Nov 03 18:12:04.881692 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:04.881658    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36d4f867-6750-43de-8cdc-76f7e76fdd44-workload-certs\") pod \"e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r\" (UID: \"36d4f867-6750-43de-8cdc-76f7e76fdd44\") " pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r"
Nov 03 18:12:04.881890 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:04.881722    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36d4f867-6750-43de-8cdc-76f7e76fdd44-istio-data\") pod \"e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r\" (UID: \"36d4f867-6750-43de-8cdc-76f7e76fdd44\") " pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r"
Nov 03 18:12:04.882045 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:04.881909    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp2x2\" (UniqueName: \"kubernetes.io/projected/36d4f867-6750-43de-8cdc-76f7e76fdd44-kube-api-access-jp2x2\") pod \"e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r\" (UID: \"36d4f867-6750-43de-8cdc-76f7e76fdd44\") " pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r"
Nov 03 18:12:04.882045 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:04.882037    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d4f867-6750-43de-8cdc-76f7e76fdd44-credential-socket\") pod \"e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r\" (UID: \"36d4f867-6750-43de-8cdc-76f7e76fdd44\") " pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r"
Nov 03 18:12:04.882174 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:04.882111    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d4f867-6750-43de-8cdc-76f7e76fdd44-workload-socket\") pod \"e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r\" (UID: \"36d4f867-6750-43de-8cdc-76f7e76fdd44\") " pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r"
Nov 03 18:12:04.882301 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:04.882274    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36d4f867-6750-43de-8cdc-76f7e76fdd44-istio-podinfo\") pod \"e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r\" (UID: \"36d4f867-6750-43de-8cdc-76f7e76fdd44\") " pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r"
Nov 03 18:12:04.882430 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:04.882395    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36d4f867-6750-43de-8cdc-76f7e76fdd44-istio-envoy\") pod \"e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r\" (UID: \"36d4f867-6750-43de-8cdc-76f7e76fdd44\") " pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r"
Nov 03 18:12:04.882522 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:04.882460    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36d4f867-6750-43de-8cdc-76f7e76fdd44-istiod-ca-cert\") pod \"e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r\" (UID: \"36d4f867-6750-43de-8cdc-76f7e76fdd44\") " pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r"
Nov 03 18:12:04.882601 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:04.882573    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36d4f867-6750-43de-8cdc-76f7e76fdd44-istio-token\") pod \"e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r\" (UID: \"36d4f867-6750-43de-8cdc-76f7e76fdd44\") " pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r"
Nov 03 18:12:04.907390 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:04.907346    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r"]
Nov 03 18:12:04.985236 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:04.985196    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36d4f867-6750-43de-8cdc-76f7e76fdd44-istio-podinfo\") pod \"e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r\" (UID: \"36d4f867-6750-43de-8cdc-76f7e76fdd44\") " pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r"
Nov 03 18:12:04.985586 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:04.985557    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36d4f867-6750-43de-8cdc-76f7e76fdd44-istio-envoy\") pod \"e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r\" (UID: \"36d4f867-6750-43de-8cdc-76f7e76fdd44\") " pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r"
Nov 03 18:12:05.052695 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:05.020697    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36d4f867-6750-43de-8cdc-76f7e76fdd44-istiod-ca-cert\") pod \"e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r\" (UID: \"36d4f867-6750-43de-8cdc-76f7e76fdd44\") " pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r"
Nov 03 18:12:05.052695 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:05.020865    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36d4f867-6750-43de-8cdc-76f7e76fdd44-istio-token\") pod \"e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r\" (UID: \"36d4f867-6750-43de-8cdc-76f7e76fdd44\") " pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r"
Nov 03 18:12:05.052695 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:05.020965    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36d4f867-6750-43de-8cdc-76f7e76fdd44-workload-certs\") pod \"e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r\" (UID: \"36d4f867-6750-43de-8cdc-76f7e76fdd44\") " pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r"
Nov 03 18:12:05.052695 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:05.021035    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36d4f867-6750-43de-8cdc-76f7e76fdd44-istio-data\") pod \"e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r\" (UID: \"36d4f867-6750-43de-8cdc-76f7e76fdd44\") " pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r"
Nov 03 18:12:05.052695 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:05.021146    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp2x2\" (UniqueName: \"kubernetes.io/projected/36d4f867-6750-43de-8cdc-76f7e76fdd44-kube-api-access-jp2x2\") pod \"e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r\" (UID: \"36d4f867-6750-43de-8cdc-76f7e76fdd44\") " pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r"
Nov 03 18:12:05.052695 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:05.021233    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d4f867-6750-43de-8cdc-76f7e76fdd44-credential-socket\") pod \"e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r\" (UID: \"36d4f867-6750-43de-8cdc-76f7e76fdd44\") " pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r"
Nov 03 18:12:05.052695 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:05.021307    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d4f867-6750-43de-8cdc-76f7e76fdd44-workload-socket\") pod \"e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r\" (UID: \"36d4f867-6750-43de-8cdc-76f7e76fdd44\") " pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r"
Nov 03 18:12:05.052695 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:05.021921    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d4f867-6750-43de-8cdc-76f7e76fdd44-workload-socket\") pod \"e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r\" (UID: \"36d4f867-6750-43de-8cdc-76f7e76fdd44\") " pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r"
Nov 03 18:12:05.052695 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:05.021982    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36d4f867-6750-43de-8cdc-76f7e76fdd44-workload-certs\") pod \"e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r\" (UID: \"36d4f867-6750-43de-8cdc-76f7e76fdd44\") " pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r"
Nov 03 18:12:05.052695 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:05.022228    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36d4f867-6750-43de-8cdc-76f7e76fdd44-istio-data\") pod \"e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r\" (UID: \"36d4f867-6750-43de-8cdc-76f7e76fdd44\") " pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r"
Nov 03 18:12:05.052695 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:05.023057    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d4f867-6750-43de-8cdc-76f7e76fdd44-credential-socket\") pod \"e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r\" (UID: \"36d4f867-6750-43de-8cdc-76f7e76fdd44\") " pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r"
Nov 03 18:12:05.075637 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:05.054017    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36d4f867-6750-43de-8cdc-76f7e76fdd44-istiod-ca-cert\") pod \"e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r\" (UID: \"36d4f867-6750-43de-8cdc-76f7e76fdd44\") " pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r"
Nov 03 18:12:05.356684 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:05.351809    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18f47428-da97-456a-8200-5f665c9f5d46" path="/var/lib/kubelet/pods/18f47428-da97-456a-8200-5f665c9f5d46/volumes"
Nov 03 18:12:05.447273 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:05.447240    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36d4f867-6750-43de-8cdc-76f7e76fdd44-istio-envoy\") pod \"e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r\" (UID: \"36d4f867-6750-43de-8cdc-76f7e76fdd44\") " pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r"
Nov 03 18:12:05.462642 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:05.448949    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36d4f867-6750-43de-8cdc-76f7e76fdd44-istio-podinfo\") pod \"e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r\" (UID: \"36d4f867-6750-43de-8cdc-76f7e76fdd44\") " pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r"
Nov 03 18:12:05.462642 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:05.449873    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp2x2\" (UniqueName: \"kubernetes.io/projected/36d4f867-6750-43de-8cdc-76f7e76fdd44-kube-api-access-jp2x2\") pod \"e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r\" (UID: \"36d4f867-6750-43de-8cdc-76f7e76fdd44\") " pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r"
Nov 03 18:12:05.470324 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:05.470249    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36d4f867-6750-43de-8cdc-76f7e76fdd44-istio-token\") pod \"e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r\" (UID: \"36d4f867-6750-43de-8cdc-76f7e76fdd44\") " pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r"
Nov 03 18:12:05.537997 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:05.537252    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r"
Nov 03 18:12:06.138633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:06.136717    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r"]
Nov 03 18:12:06.145191 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:12:06.145154    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36d4f867_6750_43de_8cdc_76f7e76fdd44.slice/crio-abcbad0ff0b07a0244721069ed8e4cbad72b81f038183f3a9f6fb50ae17fad5b WatchSource:0}: Error finding container abcbad0ff0b07a0244721069ed8e4cbad72b81f038183f3a9f6fb50ae17fad5b: Status 404 returned error can't find the container with id abcbad0ff0b07a0244721069ed8e4cbad72b81f038183f3a9f6fb50ae17fad5b
Nov 03 18:12:06.186292 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:06.186216    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:12:06.186491 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:06.186409    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:12:06.186573 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:06.186501    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:12:06.567040 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:06.566742    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r" event={"ID":"36d4f867-6750-43de-8cdc-76f7e76fdd44","Type":"ContainerStarted","Data":"16228370c5964cfa3a4a65ddaca6c939b84eebd4df14ca5f5aa8eb65090781f3"}
Nov 03 18:12:06.567040 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:06.566834    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r" event={"ID":"36d4f867-6750-43de-8cdc-76f7e76fdd44","Type":"ContainerStarted","Data":"abcbad0ff0b07a0244721069ed8e4cbad72b81f038183f3a9f6fb50ae17fad5b"}
Nov 03 18:12:06.681482 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:06.681369    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r" podStartSLOduration=2.681353795 podStartE2EDuration="2.681353795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:12:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:12:06.678850054 +0000 UTC m=+1565.695048123" watchObservedRunningTime="2026-11-03 18:12:06.681353795 +0000 UTC m=+1565.697551936"
Nov 03 18:12:07.547714 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:07.547675    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r"
Nov 03 18:12:07.555036 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:07.555009    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r"
Nov 03 18:12:07.579577 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:07.578753    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r"
Nov 03 18:12:07.579832 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:07.579805    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r"
Nov 03 18:12:08.637876 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:08.637840    5315 generic.go:356] "Generic (PLEG): container finished" podID="38acc846-2251-4000-ae98-e5a2492da6a6" containerID="3445112f581194bfc56d5fb23a22b94eff030673ff239c6b2118b93cdb1d9863" exitCode=0
Nov 03 18:12:08.638351 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:08.637921    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c" event={"ID":"38acc846-2251-4000-ae98-e5a2492da6a6","Type":"ContainerDied","Data":"3445112f581194bfc56d5fb23a22b94eff030673ff239c6b2118b93cdb1d9863"}
Nov 03 18:12:08.638351 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:08.638046    5315 scope.go:122] "RemoveContainer" containerID="4f68f393ea3076a8de212d5b8c368a7c2cfd504c1693f3bbf8e150c268a96686"
Nov 03 18:12:08.726973 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:08.726946    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c"
Nov 03 18:12:08.853505 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:08.853465    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38acc846-2251-4000-ae98-e5a2492da6a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38acc846-2251-4000-ae98-e5a2492da6a6-credential-socket\") pod \"38acc846-2251-4000-ae98-e5a2492da6a6\" (UID: \"38acc846-2251-4000-ae98-e5a2492da6a6\") "
Nov 03 18:12:08.853714 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:08.853577    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38acc846-2251-4000-ae98-e5a2492da6a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38acc846-2251-4000-ae98-e5a2492da6a6-istio-envoy\") pod \"38acc846-2251-4000-ae98-e5a2492da6a6\" (UID: \"38acc846-2251-4000-ae98-e5a2492da6a6\") "
Nov 03 18:12:08.853714 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:08.853685    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38acc846-2251-4000-ae98-e5a2492da6a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38acc846-2251-4000-ae98-e5a2492da6a6-workload-socket\") pod \"38acc846-2251-4000-ae98-e5a2492da6a6\" (UID: \"38acc846-2251-4000-ae98-e5a2492da6a6\") "
Nov 03 18:12:08.853977 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:08.853728    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38acc846-2251-4000-ae98-e5a2492da6a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38acc846-2251-4000-ae98-e5a2492da6a6-istiod-ca-cert\") pod \"38acc846-2251-4000-ae98-e5a2492da6a6\" (UID: \"38acc846-2251-4000-ae98-e5a2492da6a6\") "
Nov 03 18:12:08.853977 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:08.853814    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38acc846-2251-4000-ae98-e5a2492da6a6-credential-socket" pod "38acc846-2251-4000-ae98-e5a2492da6a6" (UID: "38acc846-2251-4000-ae98-e5a2492da6a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:08.854096 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:08.854024    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38acc846-2251-4000-ae98-e5a2492da6a6-workload-socket" pod "38acc846-2251-4000-ae98-e5a2492da6a6" (UID: "38acc846-2251-4000-ae98-e5a2492da6a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:08.854096 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:08.854046    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/38acc846-2251-4000-ae98-e5a2492da6a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38acc846-2251-4000-ae98-e5a2492da6a6-istio-podinfo\") pod \"38acc846-2251-4000-ae98-e5a2492da6a6\" (UID: \"38acc846-2251-4000-ae98-e5a2492da6a6\") "
Nov 03 18:12:08.854204 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:08.854098    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38acc846-2251-4000-ae98-e5a2492da6a6-kube-api-access-7hdxx\" (UniqueName: \"kubernetes.io/projected/38acc846-2251-4000-ae98-e5a2492da6a6-kube-api-access-7hdxx\") pod \"38acc846-2251-4000-ae98-e5a2492da6a6\" (UID: \"38acc846-2251-4000-ae98-e5a2492da6a6\") "
Nov 03 18:12:08.854204 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:08.854134    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38acc846-2251-4000-ae98-e5a2492da6a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38acc846-2251-4000-ae98-e5a2492da6a6-istio-data\") pod \"38acc846-2251-4000-ae98-e5a2492da6a6\" (UID: \"38acc846-2251-4000-ae98-e5a2492da6a6\") "
Nov 03 18:12:08.854312 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:08.854259    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38acc846-2251-4000-ae98-e5a2492da6a6-istio-token\" (UniqueName: \"kubernetes.io/projected/38acc846-2251-4000-ae98-e5a2492da6a6-istio-token\") pod \"38acc846-2251-4000-ae98-e5a2492da6a6\" (UID: \"38acc846-2251-4000-ae98-e5a2492da6a6\") "
Nov 03 18:12:08.854312 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:08.854270    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38acc846-2251-4000-ae98-e5a2492da6a6-istiod-ca-cert" pod "38acc846-2251-4000-ae98-e5a2492da6a6" (UID: "38acc846-2251-4000-ae98-e5a2492da6a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:12:08.854312 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:08.854295    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38acc846-2251-4000-ae98-e5a2492da6a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38acc846-2251-4000-ae98-e5a2492da6a6-workload-certs\") pod \"38acc846-2251-4000-ae98-e5a2492da6a6\" (UID: \"38acc846-2251-4000-ae98-e5a2492da6a6\") "
Nov 03 18:12:08.854540 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:08.854517    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38acc846-2251-4000-ae98-e5a2492da6a6-istio-data" pod "38acc846-2251-4000-ae98-e5a2492da6a6" (UID: "38acc846-2251-4000-ae98-e5a2492da6a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:08.854773 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:08.854748    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38acc846-2251-4000-ae98-e5a2492da6a6-workload-certs" pod "38acc846-2251-4000-ae98-e5a2492da6a6" (UID: "38acc846-2251-4000-ae98-e5a2492da6a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:08.855216 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:08.855188    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38acc846-2251-4000-ae98-e5a2492da6a6-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:08.855336 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:08.855225    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38acc846-2251-4000-ae98-e5a2492da6a6-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:08.855336 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:08.855241    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38acc846-2251-4000-ae98-e5a2492da6a6-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:08.855336 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:08.855256    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38acc846-2251-4000-ae98-e5a2492da6a6-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:08.855336 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:08.855272    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38acc846-2251-4000-ae98-e5a2492da6a6-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:08.856468 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:08.856434    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38acc846-2251-4000-ae98-e5a2492da6a6-istio-envoy" pod "38acc846-2251-4000-ae98-e5a2492da6a6" (UID: "38acc846-2251-4000-ae98-e5a2492da6a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:08.856705 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:08.856678    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38acc846-2251-4000-ae98-e5a2492da6a6-istio-podinfo" pod "38acc846-2251-4000-ae98-e5a2492da6a6" (UID: "38acc846-2251-4000-ae98-e5a2492da6a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:12:08.856803 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:08.856781    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38acc846-2251-4000-ae98-e5a2492da6a6-kube-api-access-7hdxx" pod "38acc846-2251-4000-ae98-e5a2492da6a6" (UID: "38acc846-2251-4000-ae98-e5a2492da6a6"). InnerVolumeSpecName "kube-api-access-7hdxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:12:08.856866 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:08.856803    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38acc846-2251-4000-ae98-e5a2492da6a6-istio-token" pod "38acc846-2251-4000-ae98-e5a2492da6a6" (UID: "38acc846-2251-4000-ae98-e5a2492da6a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:12:08.957132 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:08.957040    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38acc846-2251-4000-ae98-e5a2492da6a6-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:08.957132 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:08.957085    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hdxx\" (UniqueName: \"kubernetes.io/projected/38acc846-2251-4000-ae98-e5a2492da6a6-kube-api-access-7hdxx\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:08.957132 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:08.957105    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38acc846-2251-4000-ae98-e5a2492da6a6-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:08.957132 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:08.957119    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38acc846-2251-4000-ae98-e5a2492da6a6-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:09.650841 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:09.650806    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c"
Nov 03 18:12:09.651414 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:09.650808    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c" event={"ID":"38acc846-2251-4000-ae98-e5a2492da6a6","Type":"ContainerDied","Data":"8bcfaa5b49538f5ef285ea1f570a583104a19ebd5636498c326934a43551dc90"}
Nov 03 18:12:09.651634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:09.651477    5315 scope.go:122] "RemoveContainer" containerID="3445112f581194bfc56d5fb23a22b94eff030673ff239c6b2118b93cdb1d9863"
Nov 03 18:12:09.853423 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:09.853386    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c"]
Nov 03 18:12:09.899476 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:09.899439    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-5ffd208e-openshift-default-5cdf57cc7c-hrx8c"]
Nov 03 18:12:11.110193 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:11.110149    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38acc846-2251-4000-ae98-e5a2492da6a6" path="/var/lib/kubelet/pods/38acc846-2251-4000-ae98-e5a2492da6a6/volumes"
Nov 03 18:12:12.438637 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:12.438582    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-1720c00c/e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr"]
Nov 03 18:12:12.462345 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:12.462264    5315 cpu_manager.go:414] "containerMap: removing container" podUID="38acc846-2251-4000-ae98-e5a2492da6a6" containerName="istio-proxy"
Nov 03 18:12:12.462605 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:12.462571    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38acc846-2251-4000-ae98-e5a2492da6a6" containerName="istio-proxy"
Nov 03 18:12:12.462605 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:12.462625    5315 cpu_manager.go:414] "containerMap: removing container" podUID="38acc846-2251-4000-ae98-e5a2492da6a6" containerName="istio-proxy"
Nov 03 18:12:12.462605 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:12.462635    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38acc846-2251-4000-ae98-e5a2492da6a6" containerName="istio-proxy"
Nov 03 18:12:12.462881 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:12.462866    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="38acc846-2251-4000-ae98-e5a2492da6a6" containerName="istio-proxy"
Nov 03 18:12:12.489526 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:12.489476    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1720c00c/e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr"
Nov 03 18:12:12.505757 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:12.505727    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-1720c00c/e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr"]
Nov 03 18:12:12.523771 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:12.521300    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b7d3471-e6a0-4514-bf01-afb303808e53-dshm\") pod \"e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr\" (UID: \"8b7d3471-e6a0-4514-bf01-afb303808e53\") " pod="ai-tenant-e2e-mt-1720c00c/e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr"
Nov 03 18:12:12.523771 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:12.521479    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b7d3471-e6a0-4514-bf01-afb303808e53-home\") pod \"e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr\" (UID: \"8b7d3471-e6a0-4514-bf01-afb303808e53\") " pod="ai-tenant-e2e-mt-1720c00c/e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr"
Nov 03 18:12:12.523771 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:12.521513    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b7d3471-e6a0-4514-bf01-afb303808e53-tls-certs\") pod \"e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr\" (UID: \"8b7d3471-e6a0-4514-bf01-afb303808e53\") " pod="ai-tenant-e2e-mt-1720c00c/e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr"
Nov 03 18:12:12.523771 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:12.521673    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b7d3471-e6a0-4514-bf01-afb303808e53-model-cache\") pod \"e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr\" (UID: \"8b7d3471-e6a0-4514-bf01-afb303808e53\") " pod="ai-tenant-e2e-mt-1720c00c/e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr"
Nov 03 18:12:12.523771 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:12.521715    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2xkn\" (UniqueName: \"kubernetes.io/projected/8b7d3471-e6a0-4514-bf01-afb303808e53-kube-api-access-k2xkn\") pod \"e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr\" (UID: \"8b7d3471-e6a0-4514-bf01-afb303808e53\") " pod="ai-tenant-e2e-mt-1720c00c/e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr"
Nov 03 18:12:12.523771 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:12.521788    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b7d3471-e6a0-4514-bf01-afb303808e53-tmp-dir\") pod \"e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr\" (UID: \"8b7d3471-e6a0-4514-bf01-afb303808e53\") " pod="ai-tenant-e2e-mt-1720c00c/e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr"
Nov 03 18:12:12.710826 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:12.544165    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-1720c00c\"/\"e2e-lifecycle-model-1720c00c-kserve-self-signed-certs\""
Nov 03 18:12:12.710826 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:12.544165    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-1720c00c\"/\"default-dockercfg-cwkhk\""
Nov 03 18:12:12.710826 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:12.584269    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-1720c00c\"/\"kube-root-ca.crt\""
Nov 03 18:12:12.710826 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:12.584388    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-1720c00c\"/\"openshift-service-ca.crt\""
Nov 03 18:12:12.710826 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:12.627515    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b7d3471-e6a0-4514-bf01-afb303808e53-dshm\") pod \"e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr\" (UID: \"8b7d3471-e6a0-4514-bf01-afb303808e53\") " pod="ai-tenant-e2e-mt-1720c00c/e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr"
Nov 03 18:12:12.710826 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:12.627699    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b7d3471-e6a0-4514-bf01-afb303808e53-home\") pod \"e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr\" (UID: \"8b7d3471-e6a0-4514-bf01-afb303808e53\") " pod="ai-tenant-e2e-mt-1720c00c/e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr"
Nov 03 18:12:12.710826 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:12.627737    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b7d3471-e6a0-4514-bf01-afb303808e53-tls-certs\") pod \"e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr\" (UID: \"8b7d3471-e6a0-4514-bf01-afb303808e53\") " pod="ai-tenant-e2e-mt-1720c00c/e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr"
Nov 03 18:12:12.710826 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:12.627813    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b7d3471-e6a0-4514-bf01-afb303808e53-model-cache\") pod \"e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr\" (UID: \"8b7d3471-e6a0-4514-bf01-afb303808e53\") " pod="ai-tenant-e2e-mt-1720c00c/e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr"
Nov 03 18:12:12.710826 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:12.627853    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2xkn\" (UniqueName: \"kubernetes.io/projected/8b7d3471-e6a0-4514-bf01-afb303808e53-kube-api-access-k2xkn\") pod \"e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr\" (UID: \"8b7d3471-e6a0-4514-bf01-afb303808e53\") " pod="ai-tenant-e2e-mt-1720c00c/e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr"
Nov 03 18:12:12.710826 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:12.627885    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b7d3471-e6a0-4514-bf01-afb303808e53-tmp-dir\") pod \"e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr\" (UID: \"8b7d3471-e6a0-4514-bf01-afb303808e53\") " pod="ai-tenant-e2e-mt-1720c00c/e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr"
Nov 03 18:12:12.710826 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:12.647313    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b7d3471-e6a0-4514-bf01-afb303808e53-model-cache\") pod \"e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr\" (UID: \"8b7d3471-e6a0-4514-bf01-afb303808e53\") " pod="ai-tenant-e2e-mt-1720c00c/e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr"
Nov 03 18:12:12.710826 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:12.647592    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b7d3471-e6a0-4514-bf01-afb303808e53-home\") pod \"e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr\" (UID: \"8b7d3471-e6a0-4514-bf01-afb303808e53\") " pod="ai-tenant-e2e-mt-1720c00c/e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr"
Nov 03 18:12:12.710826 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:12.651430    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b7d3471-e6a0-4514-bf01-afb303808e53-tmp-dir\") pod \"e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr\" (UID: \"8b7d3471-e6a0-4514-bf01-afb303808e53\") " pod="ai-tenant-e2e-mt-1720c00c/e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr"
Nov 03 18:12:12.710826 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:12.654849    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b7d3471-e6a0-4514-bf01-afb303808e53-dshm\") pod \"e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr\" (UID: \"8b7d3471-e6a0-4514-bf01-afb303808e53\") " pod="ai-tenant-e2e-mt-1720c00c/e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr"
Nov 03 18:12:12.710826 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:12.662934    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b7d3471-e6a0-4514-bf01-afb303808e53-tls-certs\") pod \"e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr\" (UID: \"8b7d3471-e6a0-4514-bf01-afb303808e53\") " pod="ai-tenant-e2e-mt-1720c00c/e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr"
Nov 03 18:12:12.712515 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:12.712486    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2xkn\" (UniqueName: \"kubernetes.io/projected/8b7d3471-e6a0-4514-bf01-afb303808e53-kube-api-access-k2xkn\") pod \"e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr\" (UID: \"8b7d3471-e6a0-4514-bf01-afb303808e53\") " pod="ai-tenant-e2e-mt-1720c00c/e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr"
Nov 03 18:12:12.823966 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:12.823927    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1720c00c/e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr"
Nov 03 18:12:13.448209 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:13.448122    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-1720c00c/e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr"]
Nov 03 18:12:13.453016 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:12:13.452979    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b7d3471_e6a0_4514_bf01_afb303808e53.slice/crio-2880721b312f6820bdaed383f9034437cf086c0b5ec4ba555c947761212a5206 WatchSource:0}: Error finding container 2880721b312f6820bdaed383f9034437cf086c0b5ec4ba555c947761212a5206: Status 404 returned error can't find the container with id 2880721b312f6820bdaed383f9034437cf086c0b5ec4ba555c947761212a5206
Nov 03 18:12:13.725769 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:13.725685    5315 generic.go:356] "Generic (PLEG): container finished" podID="6ebdce76-7e0e-4916-afcd-2aaf7cc44fed" containerID="6c4ad9e24979e6ced486b18cb044721e965b37d80560cf8ccd1905831306733c" exitCode=0
Nov 03 18:12:13.725769 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:13.725729    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx" event={"ID":"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed","Type":"ContainerDied","Data":"6c4ad9e24979e6ced486b18cb044721e965b37d80560cf8ccd1905831306733c"}
Nov 03 18:12:13.726282 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:13.726257    5315 scope.go:122] "RemoveContainer" containerID="6c4ad9e24979e6ced486b18cb044721e965b37d80560cf8ccd1905831306733c"
Nov 03 18:12:13.727989 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:13.727960    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1720c00c/e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr" event={"ID":"8b7d3471-e6a0-4514-bf01-afb303808e53","Type":"ContainerStarted","Data":"b9ce47857130ff14719d41d7572ab8e0102fb6d7dcd26e99cc718bbb40a6f8e6"}
Nov 03 18:12:13.728099 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:13.728002    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1720c00c/e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr" event={"ID":"8b7d3471-e6a0-4514-bf01-afb303808e53","Type":"ContainerStarted","Data":"2880721b312f6820bdaed383f9034437cf086c0b5ec4ba555c947761212a5206"}
Nov 03 18:12:13.728339 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:13.728313    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:12:13.728399 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:13.728385    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:12:13.728446 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:13.728423    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:12:13.824644 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:13.824579    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-1720c00c/e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr"
Nov 03 18:12:13.953406 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:13.953340    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-1720c00c/e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr" podStartSLOduration=1.953318565 podStartE2EDuration="1.953318565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:12:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:12:13.94742658 +0000 UTC m=+1572.963624653" watchObservedRunningTime="2026-11-03 18:12:13.953318565 +0000 UTC m=+1572.969516638"
Nov 03 18:12:14.760103 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:14.760063    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx" event={"ID":"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed","Type":"ContainerStarted","Data":"7e1c5b05b3783b10ae91334682b73f908c0d459e6b3f486b75b5f62a1c055778"}
Nov 03 18:12:15.648795 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:15.648756    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx"
Nov 03 18:12:15.650419 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:15.650389    5315 patch_prober.go:29] interesting pod/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Nov 03 18:12:15.650547 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:15.650449    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx" podUID="6ebdce76-7e0e-4916-afcd-2aaf7cc44fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Nov 03 18:12:16.647648 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:16.644267    5315 patch_prober.go:29] interesting pod/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Nov 03 18:12:16.647648 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:16.644339    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx" podUID="6ebdce76-7e0e-4916-afcd-2aaf7cc44fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Nov 03 18:12:17.645634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:17.634983    5315 patch_prober.go:29] interesting pod/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Nov 03 18:12:17.645634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:17.635048    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx" podUID="6ebdce76-7e0e-4916-afcd-2aaf7cc44fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Nov 03 18:12:18.671193 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:18.671108    5315 patch_prober.go:29] interesting pod/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Nov 03 18:12:18.671926 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:18.671267    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx" podUID="6ebdce76-7e0e-4916-afcd-2aaf7cc44fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Nov 03 18:12:19.184337 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:19.181353    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx"
Nov 03 18:12:19.633566 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:19.633065    5315 patch_prober.go:29] interesting pod/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Nov 03 18:12:19.633566 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:19.633136    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx" podUID="6ebdce76-7e0e-4916-afcd-2aaf7cc44fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Nov 03 18:12:20.813631 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:20.808939    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx"
Nov 03 18:12:20.828785 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:20.827778    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx"
Nov 03 18:12:22.824417 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:22.824380    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-1720c00c/e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr"
Nov 03 18:12:22.838409 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:22.838383    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-1720c00c/e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr"
Nov 03 18:12:22.892701 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:22.892666    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-1720c00c/e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr"
Nov 03 18:12:24.581984 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:12:24.581938    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:12:31.899957 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:31.899906    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-1720c00c/e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr" podUID="8b7d3471-e6a0-4514-bf01-afb303808e53" containerName="main" probeResult="failure" output="Get \"https://10.133.0.167:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:12:31.934505 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:31.934458    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c" podUID="8ac59539-8ac1-466e-8d1b-b2be4c42c708" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:12:32.641638 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:32.640664    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt" podUID="7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:12:32.665771 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:32.663582    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7" podUID="69760dba-faf9-4c08-895c-ad70fe95fd49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:12:32.672937 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:32.672894    5315 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s" podUID="155e4b39-1877-4bd9-8541-5d6bd3aa0a8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:12:32.677772 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:32.677732    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l" podUID="114dcf54-e4eb-4294-a77b-604fe6d57d48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:12:32.685174 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:32.685135    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q" podUID="c974b0f1-e00b-4e4a-b193-df769068983d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:12:32.686202 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:32.686172    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s" podUID="155e4b39-1877-4bd9-8541-5d6bd3aa0a8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Nov 03 18:12:32.689205 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:32.689175    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr" podUID="df8e0b71-e876-4cea-a7ae-3532c887c660" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:12:33.423826 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:33.423776    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7" podUID="33725a51-6459-4f15-a866-e9dee41a9732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:12:33.462636 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:33.454040    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv" podUID="f6d706cb-3aa5-419c-8f6c-29347bb788b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:12:33.589735 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:33.586769    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7" podUID="09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:12:33.628867 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:33.628811    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr" podUID="08159a02-fc2e-4138-824d-be5ba3e141d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:12:33.645368 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:33.642233    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt" podUID="7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:12:33.645368 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:33.642665    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt"
Nov 03 18:12:33.670667 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:33.670590    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7" podUID="69760dba-faf9-4c08-895c-ad70fe95fd49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:12:33.671422 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:33.671400    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7"
Nov 03 18:12:33.708688 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:33.698720    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr" podUID="df8e0b71-e876-4cea-a7ae-3532c887c660" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:12:33.708688 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:33.699113    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q" podUID="c974b0f1-e00b-4e4a-b193-df769068983d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 18:12:33.708688 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:33.699399    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s" podUID="155e4b39-1877-4bd9-8541-5d6bd3aa0a8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Nov 03 18:12:33.708688 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:33.699697    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr"
Nov 03 18:12:33.708688 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:33.699777    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q"
Nov 03 18:12:33.708688 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:33.699813    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s"
Nov 03 18:12:33.716472 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:33.716429    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l" podUID="114dcf54-e4eb-4294-a77b-604fe6d57d48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:12:33.717072 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:33.717048    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l"
Nov 03 18:12:33.889854 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:33.889810    5315 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-mt-1720c00c/e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr" podUID="8b7d3471-e6a0-4514-bf01-afb303808e53" containerName="main" probeResult="failure" output="Get \"https://10.133.0.167:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:12:33.998755 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:33.891980    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-1720c00c/e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr" podUID="8b7d3471-e6a0-4514-bf01-afb303808e53" containerName="main" probeResult="failure" output="Get \"https://10.133.0.167:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:12:33.998755 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:33.892013    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c" podUID="8ac59539-8ac1-466e-8d1b-b2be4c42c708" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Nov 03 18:12:33.998755 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:33.892034    5315 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr" podUID="08159a02-fc2e-4138-824d-be5ba3e141d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": context deadline exceeded"
Nov 03 18:12:34.440731 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:34.424894    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7" podUID="33725a51-6459-4f15-a866-e9dee41a9732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:12:34.468701 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:34.465591    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7"
Nov 03 18:12:34.468701 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:34.467202    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv" podUID="f6d706cb-3aa5-419c-8f6c-29347bb788b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:12:34.490650 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:34.486260    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv"
Nov 03 18:12:34.641584 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:34.641542    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7" podUID="09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Nov 03 18:12:34.642305 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:34.642266    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr" podUID="08159a02-fc2e-4138-824d-be5ba3e141d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded"
Nov 03 18:12:34.658017 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:34.643385    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7"
Nov 03 18:12:34.658320 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:34.658300    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr"
Nov 03 18:12:34.662510 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:34.662475    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt" podUID="7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:12:34.714692 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:34.712681    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s" podUID="155e4b39-1877-4bd9-8541-5d6bd3aa0a8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:12:34.714692 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:34.713020    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7" podUID="69760dba-faf9-4c08-895c-ad70fe95fd49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/ready\": context deadline exceeded"
Nov 03 18:12:34.714692 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:34.713260    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr" podUID="df8e0b71-e876-4cea-a7ae-3532c887c660" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 18:12:34.715461 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:34.715433    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q" podUID="c974b0f1-e00b-4e4a-b193-df769068983d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 18:12:34.717687 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:34.717658    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l" podUID="114dcf54-e4eb-4294-a77b-604fe6d57d48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:12:34.934632 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:34.929543    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7"
Nov 03 18:12:35.012166 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:35.011557    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c" podUID="8ac59539-8ac1-466e-8d1b-b2be4c42c708" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:12:35.031637 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:35.029314    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-1720c00c/e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr" podUID="8b7d3471-e6a0-4514-bf01-afb303808e53" containerName="main" probeResult="failure" output="Get \"https://10.133.0.167:8000/ready\": context deadline exceeded"
Nov 03 18:12:35.031637 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:35.030772    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-1720c00c/e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr"
Nov 03 18:12:35.057722 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:35.047720    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c"
Nov 03 18:12:35.072106 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:35.072073    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e49abb84-phtcf"]
Nov 03 18:12:35.406600 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:35.398769    5315 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q" podUID="c974b0f1-e00b-4e4a-b193-df769068983d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded"
Nov 03 18:12:35.429065 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:35.429016    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7" podUID="33725a51-6459-4f15-a866-e9dee41a9732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:12:35.468187 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:35.468138    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv" podUID="f6d706cb-3aa5-419c-8f6c-29347bb788b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 18:12:35.677984 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:35.677701    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr" podUID="08159a02-fc2e-4138-824d-be5ba3e141d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded"
Nov 03 18:12:35.700449 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:35.677703    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7" podUID="09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Nov 03 18:12:35.724808 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:35.724751    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr" podUID="df8e0b71-e876-4cea-a7ae-3532c887c660" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:12:35.751634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:35.726710    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q" podUID="c974b0f1-e00b-4e4a-b193-df769068983d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:12:35.753206 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:35.727913    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s" podUID="155e4b39-1877-4bd9-8541-5d6bd3aa0a8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Nov 03 18:12:35.834670 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:35.797969    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l" podUID="114dcf54-e4eb-4294-a77b-604fe6d57d48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:12:35.834983 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:35.803283    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt"
Nov 03 18:12:35.835195 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:35.835163    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e49abb84-phtcf"]
Nov 03 18:12:35.835509 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:35.835488    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-1720c00c/e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr"
Nov 03 18:12:35.835744 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:35.835730    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7"
Nov 03 18:12:35.835958 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:35.835945    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv"
Nov 03 18:12:36.053757 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:36.046713    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c" podUID="8ac59539-8ac1-466e-8d1b-b2be4c42c708" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Nov 03 18:12:36.088797 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:36.088747    5315 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr" podUID="df8e0b71-e876-4cea-a7ae-3532c887c660" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:12:36.209125 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:36.208357    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s"
Nov 03 18:12:36.331984 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:36.331900    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l"
Nov 03 18:12:36.375039 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:36.374975    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr"
Nov 03 18:12:36.375441 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:36.375421    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7"
Nov 03 18:12:36.415600 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:36.415569    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c"
Nov 03 18:12:36.416945 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:36.416918    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr"
Nov 03 18:12:36.509496 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:36.509454    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q"
Nov 03 18:12:37.110049 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:37.110014    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccbbd645-e40b-4d27-8b93-e6d91a2d0184" path="/var/lib/kubelet/pods/ccbbd645-e40b-4d27-8b93-e6d91a2d0184/volumes"
Nov 03 18:12:43.048182 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:43.048144    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-727fe8c5-gtqh9"]
Nov 03 18:12:43.067093 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:43.067057    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-727fe8c5-gtqh9"]
Nov 03 18:12:43.110057 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:43.110021    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b48dcdb-9a6d-4e03-854f-ece61e03990e" path="/var/lib/kubelet/pods/6b48dcdb-9a6d-4e03-854f-ece61e03990e/volumes"
Nov 03 18:12:45.814682 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:45.814629    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n"]
Nov 03 18:12:45.815228 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:45.814896    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n" podUID="1e4ca152-3632-485f-94b2-4f430cbdfe4c" containerName="istio-proxy" containerID="cri-o://fc9e4e62d13aca2b8af6f9ad6220a16317402c53efa4273edf35cdb15050d371" gracePeriod=30
Nov 03 18:12:45.860484 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:45.860452    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p"]
Nov 03 18:12:45.860707 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:45.860683    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p" podUID="79050b41-0f3b-4fc4-a8e8-50316359ac33" containerName="istio-proxy" containerID="cri-o://cafbf7926af78b8aff2e8c4409d8633c9a4efb7801f5442b4f66d96444e558aa" gracePeriod=30
Nov 03 18:12:45.865380 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:45.865352    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89"]
Nov 03 18:12:45.865605 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:45.865573    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89" podUID="57d6511c-f9f2-4ca7-9bfb-6475dd17f94b" containerName="istio-proxy" containerID="cri-o://dbe840016f3b77963094a5c29481f8902ccdbaba738663cc148f0f91459ca3f7" gracePeriod=30
Nov 03 18:12:45.906593 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:45.906559    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb"]
Nov 03 18:12:45.906800 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:45.906779    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb" podUID="858e4f16-d2d2-4ba6-9e75-b81fe8ad5674" containerName="istio-proxy" containerID="cri-o://b9d849e23423046d448c22db5a82cbfa8eda6c1ae42c10d53a41141d36404331" gracePeriod=30
Nov 03 18:12:46.356621 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:46.356569    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm"]
Nov 03 18:12:46.356887 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:46.356859    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm" podUID="375f4e00-8c81-4de5-97bf-9be54ea028ed" containerName="istio-proxy" containerID="cri-o://79090806e6131c0c136e68a076d2e9ad0df849ff4976a5d41b06c0155b0be5f2" gracePeriod=30
Nov 03 18:12:46.362083 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:46.362048    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t"]
Nov 03 18:12:46.362938 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:46.362578    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t" podUID="2b453729-cac0-4839-bdfa-0da04e59c1bc" containerName="istio-proxy" containerID="cri-o://0feeb0a45b8262c496c57b031f48c2254a1e7ed1a311b3614f093f5363fa1829" gracePeriod=30
Nov 03 18:12:46.367563 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:46.367534    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9"]
Nov 03 18:12:46.367967 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:46.367937    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9" podUID="25a8498d-e23e-44a9-81fc-3170b124ad13" containerName="istio-proxy" containerID="cri-o://0aefff3fa1372a9e3b95667273a96b7ba828e9bf2c79b586bd15234b49e7aacf" gracePeriod=30
Nov 03 18:12:46.372348 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:46.372319    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82"]
Nov 03 18:12:46.372788 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:46.372713    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82" podUID="04fd7ef5-cf3c-4ed7-963d-e03294eaf327" containerName="istio-proxy" containerID="cri-o://e56a7f73492bd3b33294e91ede46de4ea9ee4b02e93ea8575c5f9dbc2f12167d" gracePeriod=30
Nov 03 18:12:46.626583 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:46.626503    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:12:46.626583 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:46.626561    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t" podUID="2b453729-cac0-4839-bdfa-0da04e59c1bc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:12:46.804923 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:46.804878    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:12:46.805094 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:46.804952    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82" podUID="04fd7ef5-cf3c-4ed7-963d-e03294eaf327" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:12:49.390334 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:49.390249    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:12:49.390334 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:49.390309    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p" podUID="79050b41-0f3b-4fc4-a8e8-50316359ac33" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:12:51.168966 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.168924    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n"
Nov 03 18:12:51.227480 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.227397    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e4ca152-3632-485f-94b2-4f430cbdfe4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e4ca152-3632-485f-94b2-4f430cbdfe4c-istio-podinfo\") pod \"1e4ca152-3632-485f-94b2-4f430cbdfe4c\" (UID: \"1e4ca152-3632-485f-94b2-4f430cbdfe4c\") "
Nov 03 18:12:51.227670 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.227577    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e4ca152-3632-485f-94b2-4f430cbdfe4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e4ca152-3632-485f-94b2-4f430cbdfe4c-credential-socket\") pod \"1e4ca152-3632-485f-94b2-4f430cbdfe4c\" (UID: \"1e4ca152-3632-485f-94b2-4f430cbdfe4c\") "
Nov 03 18:12:51.227670 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.227659    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e4ca152-3632-485f-94b2-4f430cbdfe4c-kube-api-access-wlrnr\" (UniqueName: \"kubernetes.io/projected/1e4ca152-3632-485f-94b2-4f430cbdfe4c-kube-api-access-wlrnr\") pod \"1e4ca152-3632-485f-94b2-4f430cbdfe4c\" (UID: \"1e4ca152-3632-485f-94b2-4f430cbdfe4c\") "
Nov 03 18:12:51.227813 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.227760    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e4ca152-3632-485f-94b2-4f430cbdfe4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e4ca152-3632-485f-94b2-4f430cbdfe4c-istio-data\") pod \"1e4ca152-3632-485f-94b2-4f430cbdfe4c\" (UID: \"1e4ca152-3632-485f-94b2-4f430cbdfe4c\") "
Nov 03 18:12:51.227813 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.227786    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e4ca152-3632-485f-94b2-4f430cbdfe4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e4ca152-3632-485f-94b2-4f430cbdfe4c-istio-envoy\") pod \"1e4ca152-3632-485f-94b2-4f430cbdfe4c\" (UID: \"1e4ca152-3632-485f-94b2-4f430cbdfe4c\") "
Nov 03 18:12:51.227993 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.227853    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e4ca152-3632-485f-94b2-4f430cbdfe4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e4ca152-3632-485f-94b2-4f430cbdfe4c-istiod-ca-cert\") pod \"1e4ca152-3632-485f-94b2-4f430cbdfe4c\" (UID: \"1e4ca152-3632-485f-94b2-4f430cbdfe4c\") "
Nov 03 18:12:51.227993 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.227914    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e4ca152-3632-485f-94b2-4f430cbdfe4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e4ca152-3632-485f-94b2-4f430cbdfe4c-workload-certs\") pod \"1e4ca152-3632-485f-94b2-4f430cbdfe4c\" (UID: \"1e4ca152-3632-485f-94b2-4f430cbdfe4c\") "
Nov 03 18:12:51.227993 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.227960    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e4ca152-3632-485f-94b2-4f430cbdfe4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e4ca152-3632-485f-94b2-4f430cbdfe4c-workload-socket\") pod \"1e4ca152-3632-485f-94b2-4f430cbdfe4c\" (UID: \"1e4ca152-3632-485f-94b2-4f430cbdfe4c\") "
Nov 03 18:12:51.228170 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.228066    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e4ca152-3632-485f-94b2-4f430cbdfe4c-istio-token\" (UniqueName: \"kubernetes.io/projected/1e4ca152-3632-485f-94b2-4f430cbdfe4c-istio-token\") pod \"1e4ca152-3632-485f-94b2-4f430cbdfe4c\" (UID: \"1e4ca152-3632-485f-94b2-4f430cbdfe4c\") "
Nov 03 18:12:51.228443 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.228380    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e4ca152-3632-485f-94b2-4f430cbdfe4c-istio-data" pod "1e4ca152-3632-485f-94b2-4f430cbdfe4c" (UID: "1e4ca152-3632-485f-94b2-4f430cbdfe4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:51.229121 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.229096    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e4ca152-3632-485f-94b2-4f430cbdfe4c-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:51.230005 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.229937    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e4ca152-3632-485f-94b2-4f430cbdfe4c-credential-socket" pod "1e4ca152-3632-485f-94b2-4f430cbdfe4c" (UID: "1e4ca152-3632-485f-94b2-4f430cbdfe4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:51.230257 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.230235    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e4ca152-3632-485f-94b2-4f430cbdfe4c-istiod-ca-cert" pod "1e4ca152-3632-485f-94b2-4f430cbdfe4c" (UID: "1e4ca152-3632-485f-94b2-4f430cbdfe4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:12:51.230802 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.230759    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e4ca152-3632-485f-94b2-4f430cbdfe4c-workload-socket" pod "1e4ca152-3632-485f-94b2-4f430cbdfe4c" (UID: "1e4ca152-3632-485f-94b2-4f430cbdfe4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:51.230905 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.230827    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e4ca152-3632-485f-94b2-4f430cbdfe4c-workload-certs" pod "1e4ca152-3632-485f-94b2-4f430cbdfe4c" (UID: "1e4ca152-3632-485f-94b2-4f430cbdfe4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:51.231944 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.231831    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e4ca152-3632-485f-94b2-4f430cbdfe4c-istio-podinfo" pod "1e4ca152-3632-485f-94b2-4f430cbdfe4c" (UID: "1e4ca152-3632-485f-94b2-4f430cbdfe4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:12:51.236939 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.236863    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e4ca152-3632-485f-94b2-4f430cbdfe4c-kube-api-access-wlrnr" pod "1e4ca152-3632-485f-94b2-4f430cbdfe4c" (UID: "1e4ca152-3632-485f-94b2-4f430cbdfe4c"). InnerVolumeSpecName "kube-api-access-wlrnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:12:51.238134 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.238106    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e4ca152-3632-485f-94b2-4f430cbdfe4c-istio-envoy" pod "1e4ca152-3632-485f-94b2-4f430cbdfe4c" (UID: "1e4ca152-3632-485f-94b2-4f430cbdfe4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:51.241501 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.241470    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e4ca152-3632-485f-94b2-4f430cbdfe4c-istio-token" pod "1e4ca152-3632-485f-94b2-4f430cbdfe4c" (UID: "1e4ca152-3632-485f-94b2-4f430cbdfe4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:12:51.296927 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.296896    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89"
Nov 03 18:12:51.307182 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.307151    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p"
Nov 03 18:12:51.318513 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.318488    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb"
Nov 03 18:12:51.331259 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.331225    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e4ca152-3632-485f-94b2-4f430cbdfe4c-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:51.331259 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.331256    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e4ca152-3632-485f-94b2-4f430cbdfe4c-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:51.331444 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.331273    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlrnr\" (UniqueName: \"kubernetes.io/projected/1e4ca152-3632-485f-94b2-4f430cbdfe4c-kube-api-access-wlrnr\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:51.331444 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.331289    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e4ca152-3632-485f-94b2-4f430cbdfe4c-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:51.331444 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.331301    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e4ca152-3632-485f-94b2-4f430cbdfe4c-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:51.331444 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.331313    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e4ca152-3632-485f-94b2-4f430cbdfe4c-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:51.331444 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.331325    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e4ca152-3632-485f-94b2-4f430cbdfe4c-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:51.331444 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.331338    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e4ca152-3632-485f-94b2-4f430cbdfe4c-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:51.387083 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.387044    5315 generic.go:356] "Generic (PLEG): container finished" podID="57d6511c-f9f2-4ca7-9bfb-6475dd17f94b" containerID="dbe840016f3b77963094a5c29481f8902ccdbaba738663cc148f0f91459ca3f7" exitCode=0
Nov 03 18:12:51.387273 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.387244    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89"
Nov 03 18:12:51.387530 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.387498    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89" event={"ID":"57d6511c-f9f2-4ca7-9bfb-6475dd17f94b","Type":"ContainerDied","Data":"dbe840016f3b77963094a5c29481f8902ccdbaba738663cc148f0f91459ca3f7"}
Nov 03 18:12:51.387676 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.387564    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89" event={"ID":"57d6511c-f9f2-4ca7-9bfb-6475dd17f94b","Type":"ContainerDied","Data":"c4afd48ae10bf1cc64a9ea1744c1308935e30b2df494ed6d17526328607d81e2"}
Nov 03 18:12:51.387676 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.387600    5315 scope.go:122] "RemoveContainer" containerID="dbe840016f3b77963094a5c29481f8902ccdbaba738663cc148f0f91459ca3f7"
Nov 03 18:12:51.390394 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.390340    5315 generic.go:356] "Generic (PLEG): container finished" podID="1e4ca152-3632-485f-94b2-4f430cbdfe4c" containerID="fc9e4e62d13aca2b8af6f9ad6220a16317402c53efa4273edf35cdb15050d371" exitCode=0
Nov 03 18:12:51.390544 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.390529    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n" event={"ID":"1e4ca152-3632-485f-94b2-4f430cbdfe4c","Type":"ContainerDied","Data":"fc9e4e62d13aca2b8af6f9ad6220a16317402c53efa4273edf35cdb15050d371"}
Nov 03 18:12:51.390586 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.390568    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n" event={"ID":"1e4ca152-3632-485f-94b2-4f430cbdfe4c","Type":"ContainerDied","Data":"7b1ea837003b2acfec8f1382c79898423ca89d7ec70da97b28504667dbda28ec"}
Nov 03 18:12:51.390984 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.390695    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n"
Nov 03 18:12:51.392463 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.392443    5315 generic.go:356] "Generic (PLEG): container finished" podID="ae6e0c1a-a26e-4b44-8e94-540944db8bd8" containerID="15b420fad5c451d5c774e7dea258301c1281506c0f5e05beb989a445f9b6ca1f" exitCode=0
Nov 03 18:12:51.392560 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.392517    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d" event={"ID":"ae6e0c1a-a26e-4b44-8e94-540944db8bd8","Type":"ContainerDied","Data":"15b420fad5c451d5c774e7dea258301c1281506c0f5e05beb989a445f9b6ca1f"}
Nov 03 18:12:51.392977 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.392962    5315 scope.go:122] "RemoveContainer" containerID="15b420fad5c451d5c774e7dea258301c1281506c0f5e05beb989a445f9b6ca1f"
Nov 03 18:12:51.395352 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.395332    5315 generic.go:356] "Generic (PLEG): container finished" podID="858e4f16-d2d2-4ba6-9e75-b81fe8ad5674" containerID="b9d849e23423046d448c22db5a82cbfa8eda6c1ae42c10d53a41141d36404331" exitCode=0
Nov 03 18:12:51.395467 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.395363    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb" event={"ID":"858e4f16-d2d2-4ba6-9e75-b81fe8ad5674","Type":"ContainerDied","Data":"b9d849e23423046d448c22db5a82cbfa8eda6c1ae42c10d53a41141d36404331"}
Nov 03 18:12:51.395467 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.395389    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb"
Nov 03 18:12:51.395467 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.395418    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb" event={"ID":"858e4f16-d2d2-4ba6-9e75-b81fe8ad5674","Type":"ContainerDied","Data":"1f87bfb2af9e537420c9baf9ed14bf092911aa1cd3b30ab69b32c6300a425a2b"}
Nov 03 18:12:51.397327 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.397307    5315 generic.go:356] "Generic (PLEG): container finished" podID="79050b41-0f3b-4fc4-a8e8-50316359ac33" containerID="cafbf7926af78b8aff2e8c4409d8633c9a4efb7801f5442b4f66d96444e558aa" exitCode=0
Nov 03 18:12:51.397426 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.397333    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p" event={"ID":"79050b41-0f3b-4fc4-a8e8-50316359ac33","Type":"ContainerDied","Data":"cafbf7926af78b8aff2e8c4409d8633c9a4efb7801f5442b4f66d96444e558aa"}
Nov 03 18:12:51.397426 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.397358    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p"
Nov 03 18:12:51.397426 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.397384    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p" event={"ID":"79050b41-0f3b-4fc4-a8e8-50316359ac33","Type":"ContainerDied","Data":"e82b7f630e2085cd3b199bd014ee74b728e7c96a216562d33940419fceb16b9e"}
Nov 03 18:12:51.412259 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.412224    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:12:51.412364 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.412301    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:12:51.412364 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.412337    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:12:51.432780 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.432755    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-workload-certs\") pod \"57d6511c-f9f2-4ca7-9bfb-6475dd17f94b\" (UID: \"57d6511c-f9f2-4ca7-9bfb-6475dd17f94b\") "
Nov 03 18:12:51.432918 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.432823    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79050b41-0f3b-4fc4-a8e8-50316359ac33-istio-token\" (UniqueName: \"kubernetes.io/projected/79050b41-0f3b-4fc4-a8e8-50316359ac33-istio-token\") pod \"79050b41-0f3b-4fc4-a8e8-50316359ac33\" (UID: \"79050b41-0f3b-4fc4-a8e8-50316359ac33\") "
Nov 03 18:12:51.432918 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.432900    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79050b41-0f3b-4fc4-a8e8-50316359ac33-kube-api-access-q66b5\" (UniqueName: \"kubernetes.io/projected/79050b41-0f3b-4fc4-a8e8-50316359ac33-kube-api-access-q66b5\") pod \"79050b41-0f3b-4fc4-a8e8-50316359ac33\" (UID: \"79050b41-0f3b-4fc4-a8e8-50316359ac33\") "
Nov 03 18:12:51.433031 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.432948    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-istiod-ca-cert\") pod \"57d6511c-f9f2-4ca7-9bfb-6475dd17f94b\" (UID: \"57d6511c-f9f2-4ca7-9bfb-6475dd17f94b\") "
Nov 03 18:12:51.433031 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.433003    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79050b41-0f3b-4fc4-a8e8-50316359ac33-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79050b41-0f3b-4fc4-a8e8-50316359ac33-istio-podinfo\") pod \"79050b41-0f3b-4fc4-a8e8-50316359ac33\" (UID: \"79050b41-0f3b-4fc4-a8e8-50316359ac33\") "
Nov 03 18:12:51.433134 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.433042    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-credential-socket\") pod \"858e4f16-d2d2-4ba6-9e75-b81fe8ad5674\" (UID: \"858e4f16-d2d2-4ba6-9e75-b81fe8ad5674\") "
Nov 03 18:12:51.433203 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.433178    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-istio-podinfo\") pod \"57d6511c-f9f2-4ca7-9bfb-6475dd17f94b\" (UID: \"57d6511c-f9f2-4ca7-9bfb-6475dd17f94b\") "
Nov 03 18:12:51.433278 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.433260    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-kube-api-access-lskdx\" (UniqueName: \"kubernetes.io/projected/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-kube-api-access-lskdx\") pod \"858e4f16-d2d2-4ba6-9e75-b81fe8ad5674\" (UID: \"858e4f16-d2d2-4ba6-9e75-b81fe8ad5674\") "
Nov 03 18:12:51.433339 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.433294    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-credential-socket" pod "858e4f16-d2d2-4ba6-9e75-b81fe8ad5674" (UID: "858e4f16-d2d2-4ba6-9e75-b81fe8ad5674"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:51.433391 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.433347    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79050b41-0f3b-4fc4-a8e8-50316359ac33-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79050b41-0f3b-4fc4-a8e8-50316359ac33-istio-envoy\") pod \"79050b41-0f3b-4fc4-a8e8-50316359ac33\" (UID: \"79050b41-0f3b-4fc4-a8e8-50316359ac33\") "
Nov 03 18:12:51.433447 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.433393    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-istiod-ca-cert\") pod \"858e4f16-d2d2-4ba6-9e75-b81fe8ad5674\" (UID: \"858e4f16-d2d2-4ba6-9e75-b81fe8ad5674\") "
Nov 03 18:12:51.433504 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.433458    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79050b41-0f3b-4fc4-a8e8-50316359ac33-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79050b41-0f3b-4fc4-a8e8-50316359ac33-credential-socket\") pod \"79050b41-0f3b-4fc4-a8e8-50316359ac33\" (UID: \"79050b41-0f3b-4fc4-a8e8-50316359ac33\") "
Nov 03 18:12:51.433564 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.433502    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-istio-token\" (UniqueName: \"kubernetes.io/projected/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-istio-token\") pod \"858e4f16-d2d2-4ba6-9e75-b81fe8ad5674\" (UID: \"858e4f16-d2d2-4ba6-9e75-b81fe8ad5674\") "
Nov 03 18:12:51.433564 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.433522    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-istiod-ca-cert" pod "57d6511c-f9f2-4ca7-9bfb-6475dd17f94b" (UID: "57d6511c-f9f2-4ca7-9bfb-6475dd17f94b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:12:51.433721 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.433593    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-istio-podinfo\") pod \"858e4f16-d2d2-4ba6-9e75-b81fe8ad5674\" (UID: \"858e4f16-d2d2-4ba6-9e75-b81fe8ad5674\") "
Nov 03 18:12:51.433721 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.433667    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-istio-data\") pod \"858e4f16-d2d2-4ba6-9e75-b81fe8ad5674\" (UID: \"858e4f16-d2d2-4ba6-9e75-b81fe8ad5674\") "
Nov 03 18:12:51.433721 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.433693    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-istio-token\" (UniqueName: \"kubernetes.io/projected/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-istio-token\") pod \"57d6511c-f9f2-4ca7-9bfb-6475dd17f94b\" (UID: \"57d6511c-f9f2-4ca7-9bfb-6475dd17f94b\") "
Nov 03 18:12:51.433885 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.433726    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-workload-socket\") pod \"858e4f16-d2d2-4ba6-9e75-b81fe8ad5674\" (UID: \"858e4f16-d2d2-4ba6-9e75-b81fe8ad5674\") "
Nov 03 18:12:51.433885 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.433784    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79050b41-0f3b-4fc4-a8e8-50316359ac33-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79050b41-0f3b-4fc4-a8e8-50316359ac33-workload-certs\") pod \"79050b41-0f3b-4fc4-a8e8-50316359ac33\" (UID: \"79050b41-0f3b-4fc4-a8e8-50316359ac33\") "
Nov 03 18:12:51.433885 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.433789    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79050b41-0f3b-4fc4-a8e8-50316359ac33-credential-socket" pod "79050b41-0f3b-4fc4-a8e8-50316359ac33" (UID: "79050b41-0f3b-4fc4-a8e8-50316359ac33"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:51.433885 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.433834    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-credential-socket\") pod \"57d6511c-f9f2-4ca7-9bfb-6475dd17f94b\" (UID: \"57d6511c-f9f2-4ca7-9bfb-6475dd17f94b\") "
Nov 03 18:12:51.434110 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.433897    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79050b41-0f3b-4fc4-a8e8-50316359ac33-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79050b41-0f3b-4fc4-a8e8-50316359ac33-istio-data\") pod \"79050b41-0f3b-4fc4-a8e8-50316359ac33\" (UID: \"79050b41-0f3b-4fc4-a8e8-50316359ac33\") "
Nov 03 18:12:51.434110 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.433939    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-istio-envoy\") pod \"57d6511c-f9f2-4ca7-9bfb-6475dd17f94b\" (UID: \"57d6511c-f9f2-4ca7-9bfb-6475dd17f94b\") "
Nov 03 18:12:51.434110 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.433956    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-istiod-ca-cert" pod "858e4f16-d2d2-4ba6-9e75-b81fe8ad5674" (UID: "858e4f16-d2d2-4ba6-9e75-b81fe8ad5674"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:12:51.434110 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.433982    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-workload-certs\") pod \"858e4f16-d2d2-4ba6-9e75-b81fe8ad5674\" (UID: \"858e4f16-d2d2-4ba6-9e75-b81fe8ad5674\") "
Nov 03 18:12:51.434110 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.433986    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-istio-data" pod "858e4f16-d2d2-4ba6-9e75-b81fe8ad5674" (UID: "858e4f16-d2d2-4ba6-9e75-b81fe8ad5674"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:51.434110 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.434012    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-kube-api-access-2vsft\" (UniqueName: \"kubernetes.io/projected/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-kube-api-access-2vsft\") pod \"57d6511c-f9f2-4ca7-9bfb-6475dd17f94b\" (UID: \"57d6511c-f9f2-4ca7-9bfb-6475dd17f94b\") "
Nov 03 18:12:51.434110 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.434100    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79050b41-0f3b-4fc4-a8e8-50316359ac33-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79050b41-0f3b-4fc4-a8e8-50316359ac33-workload-socket\") pod \"79050b41-0f3b-4fc4-a8e8-50316359ac33\" (UID: \"79050b41-0f3b-4fc4-a8e8-50316359ac33\") "
Nov 03 18:12:51.434537 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.434159    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-istio-envoy\") pod \"858e4f16-d2d2-4ba6-9e75-b81fe8ad5674\" (UID: \"858e4f16-d2d2-4ba6-9e75-b81fe8ad5674\") "
Nov 03 18:12:51.434537 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.434214    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79050b41-0f3b-4fc4-a8e8-50316359ac33-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79050b41-0f3b-4fc4-a8e8-50316359ac33-istiod-ca-cert\") pod \"79050b41-0f3b-4fc4-a8e8-50316359ac33\" (UID: \"79050b41-0f3b-4fc4-a8e8-50316359ac33\") "
Nov 03 18:12:51.434537 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.434248    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-istio-data\") pod \"57d6511c-f9f2-4ca7-9bfb-6475dd17f94b\" (UID: \"57d6511c-f9f2-4ca7-9bfb-6475dd17f94b\") "
Nov 03 18:12:51.434537 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.434297    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-workload-socket" pod "858e4f16-d2d2-4ba6-9e75-b81fe8ad5674" (UID: "858e4f16-d2d2-4ba6-9e75-b81fe8ad5674"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:51.434537 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.434303    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-workload-socket\") pod \"57d6511c-f9f2-4ca7-9bfb-6475dd17f94b\" (UID: \"57d6511c-f9f2-4ca7-9bfb-6475dd17f94b\") "
Nov 03 18:12:51.434537 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.434496    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-workload-socket" pod "57d6511c-f9f2-4ca7-9bfb-6475dd17f94b" (UID: "57d6511c-f9f2-4ca7-9bfb-6475dd17f94b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:51.434917 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.434546    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79050b41-0f3b-4fc4-a8e8-50316359ac33-istio-data" pod "79050b41-0f3b-4fc4-a8e8-50316359ac33" (UID: "79050b41-0f3b-4fc4-a8e8-50316359ac33"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:51.434917 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.434744    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-workload-certs" pod "57d6511c-f9f2-4ca7-9bfb-6475dd17f94b" (UID: "57d6511c-f9f2-4ca7-9bfb-6475dd17f94b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:51.435032 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.434944    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79050b41-0f3b-4fc4-a8e8-50316359ac33-workload-certs" pod "79050b41-0f3b-4fc4-a8e8-50316359ac33" (UID: "79050b41-0f3b-4fc4-a8e8-50316359ac33"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:51.435095 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.435033    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79050b41-0f3b-4fc4-a8e8-50316359ac33-workload-socket" pod "79050b41-0f3b-4fc4-a8e8-50316359ac33" (UID: "79050b41-0f3b-4fc4-a8e8-50316359ac33"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:51.435233 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.435154    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-credential-socket" pod "57d6511c-f9f2-4ca7-9bfb-6475dd17f94b" (UID: "57d6511c-f9f2-4ca7-9bfb-6475dd17f94b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:51.435514 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.435382    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79050b41-0f3b-4fc4-a8e8-50316359ac33-istiod-ca-cert" pod "79050b41-0f3b-4fc4-a8e8-50316359ac33" (UID: "79050b41-0f3b-4fc4-a8e8-50316359ac33"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:12:51.435742 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.435717    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:51.435854 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.435747    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:51.435854 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.435761    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:51.435854 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.435775    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79050b41-0f3b-4fc4-a8e8-50316359ac33-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:51.435854 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.435790    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:51.435854 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.435806    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:51.435854 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.435821    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79050b41-0f3b-4fc4-a8e8-50316359ac33-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:51.435854 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.435835    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:51.435854 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.435849    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79050b41-0f3b-4fc4-a8e8-50316359ac33-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:51.435854 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.435864    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79050b41-0f3b-4fc4-a8e8-50316359ac33-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:51.436357 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.435879    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79050b41-0f3b-4fc4-a8e8-50316359ac33-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:51.436357 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.435892    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:51.436357 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.435906    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:51.436357 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.435802    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-istio-data" pod "57d6511c-f9f2-4ca7-9bfb-6475dd17f94b" (UID: "57d6511c-f9f2-4ca7-9bfb-6475dd17f94b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:51.436357 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.435928    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-workload-certs" pod "858e4f16-d2d2-4ba6-9e75-b81fe8ad5674" (UID: "858e4f16-d2d2-4ba6-9e75-b81fe8ad5674"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:51.436980 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.436942    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-istio-envoy" pod "57d6511c-f9f2-4ca7-9bfb-6475dd17f94b" (UID: "57d6511c-f9f2-4ca7-9bfb-6475dd17f94b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:51.437386 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.437357    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-istio-token" pod "57d6511c-f9f2-4ca7-9bfb-6475dd17f94b" (UID: "57d6511c-f9f2-4ca7-9bfb-6475dd17f94b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:12:51.437836 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.437805    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-istio-podinfo" pod "57d6511c-f9f2-4ca7-9bfb-6475dd17f94b" (UID: "57d6511c-f9f2-4ca7-9bfb-6475dd17f94b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:12:51.437928 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.437832    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79050b41-0f3b-4fc4-a8e8-50316359ac33-istio-envoy" pod "79050b41-0f3b-4fc4-a8e8-50316359ac33" (UID: "79050b41-0f3b-4fc4-a8e8-50316359ac33"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:51.437928 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.437814    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79050b41-0f3b-4fc4-a8e8-50316359ac33-istio-podinfo" pod "79050b41-0f3b-4fc4-a8e8-50316359ac33" (UID: "79050b41-0f3b-4fc4-a8e8-50316359ac33"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:12:51.437928 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.437843    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79050b41-0f3b-4fc4-a8e8-50316359ac33-istio-token" pod "79050b41-0f3b-4fc4-a8e8-50316359ac33" (UID: "79050b41-0f3b-4fc4-a8e8-50316359ac33"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:12:51.437928 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.437879    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79050b41-0f3b-4fc4-a8e8-50316359ac33-kube-api-access-q66b5" pod "79050b41-0f3b-4fc4-a8e8-50316359ac33" (UID: "79050b41-0f3b-4fc4-a8e8-50316359ac33"). InnerVolumeSpecName "kube-api-access-q66b5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:12:51.438230 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.438208    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-kube-api-access-lskdx" pod "858e4f16-d2d2-4ba6-9e75-b81fe8ad5674" (UID: "858e4f16-d2d2-4ba6-9e75-b81fe8ad5674"). InnerVolumeSpecName "kube-api-access-lskdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:12:51.438287 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.438221    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-istio-envoy" pod "858e4f16-d2d2-4ba6-9e75-b81fe8ad5674" (UID: "858e4f16-d2d2-4ba6-9e75-b81fe8ad5674"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:51.438352 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.438329    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-istio-podinfo" pod "858e4f16-d2d2-4ba6-9e75-b81fe8ad5674" (UID: "858e4f16-d2d2-4ba6-9e75-b81fe8ad5674"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:12:51.438725 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.438706    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-kube-api-access-2vsft" pod "57d6511c-f9f2-4ca7-9bfb-6475dd17f94b" (UID: "57d6511c-f9f2-4ca7-9bfb-6475dd17f94b"). InnerVolumeSpecName "kube-api-access-2vsft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:12:51.439244 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.439224    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-istio-token" pod "858e4f16-d2d2-4ba6-9e75-b81fe8ad5674" (UID: "858e4f16-d2d2-4ba6-9e75-b81fe8ad5674"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:12:51.573518 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.542246    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lskdx\" (UniqueName: \"kubernetes.io/projected/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-kube-api-access-lskdx\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:51.573518 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.542289    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79050b41-0f3b-4fc4-a8e8-50316359ac33-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:51.573518 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.542305    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:51.573518 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.542320    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:51.573518 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.542335    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:51.573518 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.542349    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:51.573518 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.542365    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:51.573518 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.542380    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vsft\" (UniqueName: \"kubernetes.io/projected/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-kube-api-access-2vsft\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:51.573518 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.542395    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:51.573518 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.542407    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:51.573518 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.542420    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79050b41-0f3b-4fc4-a8e8-50316359ac33-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:51.573518 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.542434    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q66b5\" (UniqueName: \"kubernetes.io/projected/79050b41-0f3b-4fc4-a8e8-50316359ac33-kube-api-access-q66b5\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:51.573518 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.542447    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79050b41-0f3b-4fc4-a8e8-50316359ac33-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:51.573518 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.542460    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:51.593818 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.593286    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 03 18:12:51.593818 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.593346    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm" podUID="375f4e00-8c81-4de5-97bf-9be54ea028ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 03 18:12:51.598746 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.598721    5315 scope.go:122] "RemoveContainer" containerID="dbe840016f3b77963094a5c29481f8902ccdbaba738663cc148f0f91459ca3f7"
Nov 03 18:12:51.599121 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:12:51.599092    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbe840016f3b77963094a5c29481f8902ccdbaba738663cc148f0f91459ca3f7\": container with ID starting with dbe840016f3b77963094a5c29481f8902ccdbaba738663cc148f0f91459ca3f7 not found: ID does not exist" containerID="dbe840016f3b77963094a5c29481f8902ccdbaba738663cc148f0f91459ca3f7"
Nov 03 18:12:51.599218 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.599130    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbe840016f3b77963094a5c29481f8902ccdbaba738663cc148f0f91459ca3f7"} err="failed to get container status \"dbe840016f3b77963094a5c29481f8902ccdbaba738663cc148f0f91459ca3f7\": rpc error: code = NotFound desc = could not find container \"dbe840016f3b77963094a5c29481f8902ccdbaba738663cc148f0f91459ca3f7\": container with ID starting with dbe840016f3b77963094a5c29481f8902ccdbaba738663cc148f0f91459ca3f7 not found: ID does not exist"
Nov 03 18:12:51.599218 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.599150    5315 scope.go:122] "RemoveContainer" containerID="fc9e4e62d13aca2b8af6f9ad6220a16317402c53efa4273edf35cdb15050d371"
Nov 03 18:12:51.617412 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.617239    5315 scope.go:122] "RemoveContainer" containerID="fc9e4e62d13aca2b8af6f9ad6220a16317402c53efa4273edf35cdb15050d371"
Nov 03 18:12:51.622119 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:12:51.619949    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc9e4e62d13aca2b8af6f9ad6220a16317402c53efa4273edf35cdb15050d371\": container with ID starting with fc9e4e62d13aca2b8af6f9ad6220a16317402c53efa4273edf35cdb15050d371 not found: ID does not exist" containerID="fc9e4e62d13aca2b8af6f9ad6220a16317402c53efa4273edf35cdb15050d371"
Nov 03 18:12:51.622119 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.619998    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc9e4e62d13aca2b8af6f9ad6220a16317402c53efa4273edf35cdb15050d371"} err="failed to get container status \"fc9e4e62d13aca2b8af6f9ad6220a16317402c53efa4273edf35cdb15050d371\": rpc error: code = NotFound desc = could not find container \"fc9e4e62d13aca2b8af6f9ad6220a16317402c53efa4273edf35cdb15050d371\": container with ID starting with fc9e4e62d13aca2b8af6f9ad6220a16317402c53efa4273edf35cdb15050d371 not found: ID does not exist"
Nov 03 18:12:51.622119 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.620025    5315 scope.go:122] "RemoveContainer" containerID="b9d849e23423046d448c22db5a82cbfa8eda6c1ae42c10d53a41141d36404331"
Nov 03 18:12:51.640734 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.637794    5315 scope.go:122] "RemoveContainer" containerID="b9d849e23423046d448c22db5a82cbfa8eda6c1ae42c10d53a41141d36404331"
Nov 03 18:12:51.640734 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:12:51.639225    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9d849e23423046d448c22db5a82cbfa8eda6c1ae42c10d53a41141d36404331\": container with ID starting with b9d849e23423046d448c22db5a82cbfa8eda6c1ae42c10d53a41141d36404331 not found: ID does not exist" containerID="b9d849e23423046d448c22db5a82cbfa8eda6c1ae42c10d53a41141d36404331"
Nov 03 18:12:51.640734 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.639266    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9d849e23423046d448c22db5a82cbfa8eda6c1ae42c10d53a41141d36404331"} err="failed to get container status \"b9d849e23423046d448c22db5a82cbfa8eda6c1ae42c10d53a41141d36404331\": rpc error: code = NotFound desc = could not find container \"b9d849e23423046d448c22db5a82cbfa8eda6c1ae42c10d53a41141d36404331\": container with ID starting with b9d849e23423046d448c22db5a82cbfa8eda6c1ae42c10d53a41141d36404331 not found: ID does not exist"
Nov 03 18:12:51.640734 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.639293    5315 scope.go:122] "RemoveContainer" containerID="cafbf7926af78b8aff2e8c4409d8633c9a4efb7801f5442b4f66d96444e558aa"
Nov 03 18:12:51.659201 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.657837    5315 scope.go:122] "RemoveContainer" containerID="cafbf7926af78b8aff2e8c4409d8633c9a4efb7801f5442b4f66d96444e558aa"
Nov 03 18:12:51.659201 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:12:51.658349    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cafbf7926af78b8aff2e8c4409d8633c9a4efb7801f5442b4f66d96444e558aa\": container with ID starting with cafbf7926af78b8aff2e8c4409d8633c9a4efb7801f5442b4f66d96444e558aa not found: ID does not exist" containerID="cafbf7926af78b8aff2e8c4409d8633c9a4efb7801f5442b4f66d96444e558aa"
Nov 03 18:12:51.659201 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.658388    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cafbf7926af78b8aff2e8c4409d8633c9a4efb7801f5442b4f66d96444e558aa"} err="failed to get container status \"cafbf7926af78b8aff2e8c4409d8633c9a4efb7801f5442b4f66d96444e558aa\": rpc error: code = NotFound desc = could not find container \"cafbf7926af78b8aff2e8c4409d8633c9a4efb7801f5442b4f66d96444e558aa\": container with ID starting with cafbf7926af78b8aff2e8c4409d8633c9a4efb7801f5442b4f66d96444e558aa not found: ID does not exist"
Nov 03 18:12:51.883398 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.883369    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm"
Nov 03 18:12:51.965662 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.965629    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9"
Nov 03 18:12:51.966595 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.966569    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/375f4e00-8c81-4de5-97bf-9be54ea028ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/375f4e00-8c81-4de5-97bf-9be54ea028ed-workload-socket\") pod \"375f4e00-8c81-4de5-97bf-9be54ea028ed\" (UID: \"375f4e00-8c81-4de5-97bf-9be54ea028ed\") "
Nov 03 18:12:51.968975 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.967374    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/375f4e00-8c81-4de5-97bf-9be54ea028ed-workload-socket" pod "375f4e00-8c81-4de5-97bf-9be54ea028ed" (UID: "375f4e00-8c81-4de5-97bf-9be54ea028ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:51.973008 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.972984    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/375f4e00-8c81-4de5-97bf-9be54ea028ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/375f4e00-8c81-4de5-97bf-9be54ea028ed-credential-socket\") pod \"375f4e00-8c81-4de5-97bf-9be54ea028ed\" (UID: \"375f4e00-8c81-4de5-97bf-9be54ea028ed\") "
Nov 03 18:12:51.973229 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.973212    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/375f4e00-8c81-4de5-97bf-9be54ea028ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/375f4e00-8c81-4de5-97bf-9be54ea028ed-istio-data\") pod \"375f4e00-8c81-4de5-97bf-9be54ea028ed\" (UID: \"375f4e00-8c81-4de5-97bf-9be54ea028ed\") "
Nov 03 18:12:51.973732 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.973437    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t"
Nov 03 18:12:51.973732 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.973291    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/375f4e00-8c81-4de5-97bf-9be54ea028ed-credential-socket" pod "375f4e00-8c81-4de5-97bf-9be54ea028ed" (UID: "375f4e00-8c81-4de5-97bf-9be54ea028ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:51.973732 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.973552    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/375f4e00-8c81-4de5-97bf-9be54ea028ed-istio-data" pod "375f4e00-8c81-4de5-97bf-9be54ea028ed" (UID: "375f4e00-8c81-4de5-97bf-9be54ea028ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:51.974182 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.974156    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/375f4e00-8c81-4de5-97bf-9be54ea028ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/375f4e00-8c81-4de5-97bf-9be54ea028ed-istiod-ca-cert\") pod \"375f4e00-8c81-4de5-97bf-9be54ea028ed\" (UID: \"375f4e00-8c81-4de5-97bf-9be54ea028ed\") "
Nov 03 18:12:51.974671 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.974569    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/375f4e00-8c81-4de5-97bf-9be54ea028ed-istiod-ca-cert" pod "375f4e00-8c81-4de5-97bf-9be54ea028ed" (UID: "375f4e00-8c81-4de5-97bf-9be54ea028ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:12:51.974874 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.974854    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/375f4e00-8c81-4de5-97bf-9be54ea028ed-kube-api-access-tn444\" (UniqueName: \"kubernetes.io/projected/375f4e00-8c81-4de5-97bf-9be54ea028ed-kube-api-access-tn444\") pod \"375f4e00-8c81-4de5-97bf-9be54ea028ed\" (UID: \"375f4e00-8c81-4de5-97bf-9be54ea028ed\") "
Nov 03 18:12:51.977683 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.977664    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/375f4e00-8c81-4de5-97bf-9be54ea028ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/375f4e00-8c81-4de5-97bf-9be54ea028ed-istio-envoy\") pod \"375f4e00-8c81-4de5-97bf-9be54ea028ed\" (UID: \"375f4e00-8c81-4de5-97bf-9be54ea028ed\") "
Nov 03 18:12:51.977920 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.977901    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/375f4e00-8c81-4de5-97bf-9be54ea028ed-istio-token\" (UniqueName: \"kubernetes.io/projected/375f4e00-8c81-4de5-97bf-9be54ea028ed-istio-token\") pod \"375f4e00-8c81-4de5-97bf-9be54ea028ed\" (UID: \"375f4e00-8c81-4de5-97bf-9be54ea028ed\") "
Nov 03 18:12:51.978152 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.978131    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/375f4e00-8c81-4de5-97bf-9be54ea028ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/375f4e00-8c81-4de5-97bf-9be54ea028ed-istio-podinfo\") pod \"375f4e00-8c81-4de5-97bf-9be54ea028ed\" (UID: \"375f4e00-8c81-4de5-97bf-9be54ea028ed\") "
Nov 03 18:12:51.978340 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.978308    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/375f4e00-8c81-4de5-97bf-9be54ea028ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/375f4e00-8c81-4de5-97bf-9be54ea028ed-workload-certs\") pod \"375f4e00-8c81-4de5-97bf-9be54ea028ed\" (UID: \"375f4e00-8c81-4de5-97bf-9be54ea028ed\") "
Nov 03 18:12:51.979818 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.979788    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/375f4e00-8c81-4de5-97bf-9be54ea028ed-istio-envoy" pod "375f4e00-8c81-4de5-97bf-9be54ea028ed" (UID: "375f4e00-8c81-4de5-97bf-9be54ea028ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:51.980163 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.980085    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/375f4e00-8c81-4de5-97bf-9be54ea028ed-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:51.980163 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.980128    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/375f4e00-8c81-4de5-97bf-9be54ea028ed-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:51.980163 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.980145    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/375f4e00-8c81-4de5-97bf-9be54ea028ed-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:51.980163 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.980159    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/375f4e00-8c81-4de5-97bf-9be54ea028ed-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:51.980163 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.980174    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/375f4e00-8c81-4de5-97bf-9be54ea028ed-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:51.980500 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.980468    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/375f4e00-8c81-4de5-97bf-9be54ea028ed-workload-certs" pod "375f4e00-8c81-4de5-97bf-9be54ea028ed" (UID: "375f4e00-8c81-4de5-97bf-9be54ea028ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:51.981575 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.981548    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/375f4e00-8c81-4de5-97bf-9be54ea028ed-istio-podinfo" pod "375f4e00-8c81-4de5-97bf-9be54ea028ed" (UID: "375f4e00-8c81-4de5-97bf-9be54ea028ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:12:51.981753 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.981732    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/375f4e00-8c81-4de5-97bf-9be54ea028ed-kube-api-access-tn444" pod "375f4e00-8c81-4de5-97bf-9be54ea028ed" (UID: "375f4e00-8c81-4de5-97bf-9be54ea028ed"). InnerVolumeSpecName "kube-api-access-tn444". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:12:51.982514 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.982494    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82"
Nov 03 18:12:51.983731 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.983703    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/375f4e00-8c81-4de5-97bf-9be54ea028ed-istio-token" pod "375f4e00-8c81-4de5-97bf-9be54ea028ed" (UID: "375f4e00-8c81-4de5-97bf-9be54ea028ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:12:51.999829 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:51.999764    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n"]
Nov 03 18:12:52.013654 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.013598    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-cjp6n"]
Nov 03 18:12:52.081578 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.081537    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b453729-cac0-4839-bdfa-0da04e59c1bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b453729-cac0-4839-bdfa-0da04e59c1bc-workload-certs\") pod \"2b453729-cac0-4839-bdfa-0da04e59c1bc\" (UID: \"2b453729-cac0-4839-bdfa-0da04e59c1bc\") "
Nov 03 18:12:52.081776 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.081596    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25a8498d-e23e-44a9-81fc-3170b124ad13-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25a8498d-e23e-44a9-81fc-3170b124ad13-istio-envoy\") pod \"25a8498d-e23e-44a9-81fc-3170b124ad13\" (UID: \"25a8498d-e23e-44a9-81fc-3170b124ad13\") "
Nov 03 18:12:52.081776 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.081661    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25a8498d-e23e-44a9-81fc-3170b124ad13-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a8498d-e23e-44a9-81fc-3170b124ad13-credential-socket\") pod \"25a8498d-e23e-44a9-81fc-3170b124ad13\" (UID: \"25a8498d-e23e-44a9-81fc-3170b124ad13\") "
Nov 03 18:12:52.081776 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.081697    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b453729-cac0-4839-bdfa-0da04e59c1bc-kube-api-access-f55pm\" (UniqueName: \"kubernetes.io/projected/2b453729-cac0-4839-bdfa-0da04e59c1bc-kube-api-access-f55pm\") pod \"2b453729-cac0-4839-bdfa-0da04e59c1bc\" (UID: \"2b453729-cac0-4839-bdfa-0da04e59c1bc\") "
Nov 03 18:12:52.081776 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.081738    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25a8498d-e23e-44a9-81fc-3170b124ad13-kube-api-access-rp2d5\" (UniqueName: \"kubernetes.io/projected/25a8498d-e23e-44a9-81fc-3170b124ad13-kube-api-access-rp2d5\") pod \"25a8498d-e23e-44a9-81fc-3170b124ad13\" (UID: \"25a8498d-e23e-44a9-81fc-3170b124ad13\") "
Nov 03 18:12:52.082022 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.081779    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-istio-data\") pod \"04fd7ef5-cf3c-4ed7-963d-e03294eaf327\" (UID: \"04fd7ef5-cf3c-4ed7-963d-e03294eaf327\") "
Nov 03 18:12:52.082022 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.081871    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b453729-cac0-4839-bdfa-0da04e59c1bc-workload-certs" pod "2b453729-cac0-4839-bdfa-0da04e59c1bc" (UID: "2b453729-cac0-4839-bdfa-0da04e59c1bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:52.082022 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.081888    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b453729-cac0-4839-bdfa-0da04e59c1bc-istio-token\" (UniqueName: \"kubernetes.io/projected/2b453729-cac0-4839-bdfa-0da04e59c1bc-istio-token\") pod \"2b453729-cac0-4839-bdfa-0da04e59c1bc\" (UID: \"2b453729-cac0-4839-bdfa-0da04e59c1bc\") "
Nov 03 18:12:52.082180 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.082133    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-istio-data" pod "04fd7ef5-cf3c-4ed7-963d-e03294eaf327" (UID: "04fd7ef5-cf3c-4ed7-963d-e03294eaf327"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:52.082180 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.082166    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a8498d-e23e-44a9-81fc-3170b124ad13-credential-socket" pod "25a8498d-e23e-44a9-81fc-3170b124ad13" (UID: "25a8498d-e23e-44a9-81fc-3170b124ad13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:52.082327 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.082304    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b453729-cac0-4839-bdfa-0da04e59c1bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b453729-cac0-4839-bdfa-0da04e59c1bc-istiod-ca-cert\") pod \"2b453729-cac0-4839-bdfa-0da04e59c1bc\" (UID: \"2b453729-cac0-4839-bdfa-0da04e59c1bc\") "
Nov 03 18:12:52.082400 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.082383    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25a8498d-e23e-44a9-81fc-3170b124ad13-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25a8498d-e23e-44a9-81fc-3170b124ad13-workload-certs\") pod \"25a8498d-e23e-44a9-81fc-3170b124ad13\" (UID: \"25a8498d-e23e-44a9-81fc-3170b124ad13\") "
Nov 03 18:12:52.082513 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.082498    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25a8498d-e23e-44a9-81fc-3170b124ad13-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25a8498d-e23e-44a9-81fc-3170b124ad13-istiod-ca-cert\") pod \"25a8498d-e23e-44a9-81fc-3170b124ad13\" (UID: \"25a8498d-e23e-44a9-81fc-3170b124ad13\") "
Nov 03 18:12:52.082581 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.082548    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-istio-podinfo\") pod \"04fd7ef5-cf3c-4ed7-963d-e03294eaf327\" (UID: \"04fd7ef5-cf3c-4ed7-963d-e03294eaf327\") "
Nov 03 18:12:52.082711 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.082577    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-workload-socket\") pod \"04fd7ef5-cf3c-4ed7-963d-e03294eaf327\" (UID: \"04fd7ef5-cf3c-4ed7-963d-e03294eaf327\") "
Nov 03 18:12:52.082711 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.082646    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-istiod-ca-cert\") pod \"04fd7ef5-cf3c-4ed7-963d-e03294eaf327\" (UID: \"04fd7ef5-cf3c-4ed7-963d-e03294eaf327\") "
Nov 03 18:12:52.082711 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.082699    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b453729-cac0-4839-bdfa-0da04e59c1bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b453729-cac0-4839-bdfa-0da04e59c1bc-credential-socket\") pod \"2b453729-cac0-4839-bdfa-0da04e59c1bc\" (UID: \"2b453729-cac0-4839-bdfa-0da04e59c1bc\") "
Nov 03 18:12:52.082889 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.082743    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-istio-envoy\") pod \"04fd7ef5-cf3c-4ed7-963d-e03294eaf327\" (UID: \"04fd7ef5-cf3c-4ed7-963d-e03294eaf327\") "
Nov 03 18:12:52.082889 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.082774    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25a8498d-e23e-44a9-81fc-3170b124ad13-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a8498d-e23e-44a9-81fc-3170b124ad13-workload-socket\") pod \"25a8498d-e23e-44a9-81fc-3170b124ad13\" (UID: \"25a8498d-e23e-44a9-81fc-3170b124ad13\") "
Nov 03 18:12:52.082889 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.082791    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b453729-cac0-4839-bdfa-0da04e59c1bc-istiod-ca-cert" pod "2b453729-cac0-4839-bdfa-0da04e59c1bc" (UID: "2b453729-cac0-4839-bdfa-0da04e59c1bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:12:52.082889 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.082806    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b453729-cac0-4839-bdfa-0da04e59c1bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b453729-cac0-4839-bdfa-0da04e59c1bc-istio-podinfo\") pod \"2b453729-cac0-4839-bdfa-0da04e59c1bc\" (UID: \"2b453729-cac0-4839-bdfa-0da04e59c1bc\") "
Nov 03 18:12:52.082889 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.082858    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25a8498d-e23e-44a9-81fc-3170b124ad13-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25a8498d-e23e-44a9-81fc-3170b124ad13-istio-podinfo\") pod \"25a8498d-e23e-44a9-81fc-3170b124ad13\" (UID: \"25a8498d-e23e-44a9-81fc-3170b124ad13\") "
Nov 03 18:12:52.083152 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.082907    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-credential-socket\") pod \"04fd7ef5-cf3c-4ed7-963d-e03294eaf327\" (UID: \"04fd7ef5-cf3c-4ed7-963d-e03294eaf327\") "
Nov 03 18:12:52.083152 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.083019    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-istio-token\" (UniqueName: \"kubernetes.io/projected/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-istio-token\") pod \"04fd7ef5-cf3c-4ed7-963d-e03294eaf327\" (UID: \"04fd7ef5-cf3c-4ed7-963d-e03294eaf327\") "
Nov 03 18:12:52.083152 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.083034    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b453729-cac0-4839-bdfa-0da04e59c1bc-credential-socket" pod "2b453729-cac0-4839-bdfa-0da04e59c1bc" (UID: "2b453729-cac0-4839-bdfa-0da04e59c1bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:52.083152 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.083069    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-workload-socket" pod "04fd7ef5-cf3c-4ed7-963d-e03294eaf327" (UID: "04fd7ef5-cf3c-4ed7-963d-e03294eaf327"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:52.083152 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.083076    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b453729-cac0-4839-bdfa-0da04e59c1bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b453729-cac0-4839-bdfa-0da04e59c1bc-workload-socket\") pod \"2b453729-cac0-4839-bdfa-0da04e59c1bc\" (UID: \"2b453729-cac0-4839-bdfa-0da04e59c1bc\") "
Nov 03 18:12:52.083152 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.083143    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25a8498d-e23e-44a9-81fc-3170b124ad13-istio-token\" (UniqueName: \"kubernetes.io/projected/25a8498d-e23e-44a9-81fc-3170b124ad13-istio-token\") pod \"25a8498d-e23e-44a9-81fc-3170b124ad13\" (UID: \"25a8498d-e23e-44a9-81fc-3170b124ad13\") "
Nov 03 18:12:52.083461 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.083263    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-workload-certs\") pod \"04fd7ef5-cf3c-4ed7-963d-e03294eaf327\" (UID: \"04fd7ef5-cf3c-4ed7-963d-e03294eaf327\") "
Nov 03 18:12:52.083461 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.083281    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b453729-cac0-4839-bdfa-0da04e59c1bc-workload-socket" pod "2b453729-cac0-4839-bdfa-0da04e59c1bc" (UID: "2b453729-cac0-4839-bdfa-0da04e59c1bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:52.083461 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.083310    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b453729-cac0-4839-bdfa-0da04e59c1bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b453729-cac0-4839-bdfa-0da04e59c1bc-istio-data\") pod \"2b453729-cac0-4839-bdfa-0da04e59c1bc\" (UID: \"2b453729-cac0-4839-bdfa-0da04e59c1bc\") "
Nov 03 18:12:52.083461 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.083370    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25a8498d-e23e-44a9-81fc-3170b124ad13-istiod-ca-cert" pod "25a8498d-e23e-44a9-81fc-3170b124ad13" (UID: "25a8498d-e23e-44a9-81fc-3170b124ad13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:12:52.083461 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.083387    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25a8498d-e23e-44a9-81fc-3170b124ad13-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25a8498d-e23e-44a9-81fc-3170b124ad13-istio-data\") pod \"25a8498d-e23e-44a9-81fc-3170b124ad13\" (UID: \"25a8498d-e23e-44a9-81fc-3170b124ad13\") "
Nov 03 18:12:52.083461 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.083417    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a8498d-e23e-44a9-81fc-3170b124ad13-workload-certs" pod "25a8498d-e23e-44a9-81fc-3170b124ad13" (UID: "25a8498d-e23e-44a9-81fc-3170b124ad13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:52.083461 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.083424    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b453729-cac0-4839-bdfa-0da04e59c1bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b453729-cac0-4839-bdfa-0da04e59c1bc-istio-envoy\") pod \"2b453729-cac0-4839-bdfa-0da04e59c1bc\" (UID: \"2b453729-cac0-4839-bdfa-0da04e59c1bc\") "
Nov 03 18:12:52.083896 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.083490    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-kube-api-access-klgts\" (UniqueName: \"kubernetes.io/projected/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-kube-api-access-klgts\") pod \"04fd7ef5-cf3c-4ed7-963d-e03294eaf327\" (UID: \"04fd7ef5-cf3c-4ed7-963d-e03294eaf327\") "
Nov 03 18:12:52.083896 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.083501    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b453729-cac0-4839-bdfa-0da04e59c1bc-istio-data" pod "2b453729-cac0-4839-bdfa-0da04e59c1bc" (UID: "2b453729-cac0-4839-bdfa-0da04e59c1bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:52.083896 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.083481    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-istiod-ca-cert" pod "04fd7ef5-cf3c-4ed7-963d-e03294eaf327" (UID: "04fd7ef5-cf3c-4ed7-963d-e03294eaf327"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:12:52.083896 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.083726    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-workload-certs" pod "04fd7ef5-cf3c-4ed7-963d-e03294eaf327" (UID: "04fd7ef5-cf3c-4ed7-963d-e03294eaf327"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:52.084557 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.084534    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25a8498d-e23e-44a9-81fc-3170b124ad13-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:52.084557 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.084558    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/375f4e00-8c81-4de5-97bf-9be54ea028ed-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:52.084775 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.084569    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/375f4e00-8c81-4de5-97bf-9be54ea028ed-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:52.084775 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.084578    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25a8498d-e23e-44a9-81fc-3170b124ad13-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:52.084775 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.084594    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/375f4e00-8c81-4de5-97bf-9be54ea028ed-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:52.084775 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.084643    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:52.084775 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.084655    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:52.084775 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.084666    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b453729-cac0-4839-bdfa-0da04e59c1bc-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:52.084775 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.084680    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b453729-cac0-4839-bdfa-0da04e59c1bc-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:52.084775 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.084691    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:52.084775 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.084700    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b453729-cac0-4839-bdfa-0da04e59c1bc-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:52.084775 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.084708    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b453729-cac0-4839-bdfa-0da04e59c1bc-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:52.084775 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.084717    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn444\" (UniqueName: \"kubernetes.io/projected/375f4e00-8c81-4de5-97bf-9be54ea028ed-kube-api-access-tn444\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:52.084775 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.084730    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a8498d-e23e-44a9-81fc-3170b124ad13-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:52.084775 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.084743    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:52.084775 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.084756    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b453729-cac0-4839-bdfa-0da04e59c1bc-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:52.085461 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.085412    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a8498d-e23e-44a9-81fc-3170b124ad13-workload-socket" pod "25a8498d-e23e-44a9-81fc-3170b124ad13" (UID: "25a8498d-e23e-44a9-81fc-3170b124ad13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:52.085520 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.085486    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a8498d-e23e-44a9-81fc-3170b124ad13-istio-envoy" pod "25a8498d-e23e-44a9-81fc-3170b124ad13" (UID: "25a8498d-e23e-44a9-81fc-3170b124ad13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:52.085580 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.085541    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a8498d-e23e-44a9-81fc-3170b124ad13-istio-data" pod "25a8498d-e23e-44a9-81fc-3170b124ad13" (UID: "25a8498d-e23e-44a9-81fc-3170b124ad13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:52.087572 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.087551    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b453729-cac0-4839-bdfa-0da04e59c1bc-istio-podinfo" pod "2b453729-cac0-4839-bdfa-0da04e59c1bc" (UID: "2b453729-cac0-4839-bdfa-0da04e59c1bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:12:52.087572 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.087561    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-istio-envoy" pod "04fd7ef5-cf3c-4ed7-963d-e03294eaf327" (UID: "04fd7ef5-cf3c-4ed7-963d-e03294eaf327"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:52.087946 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.087923    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-credential-socket" pod "04fd7ef5-cf3c-4ed7-963d-e03294eaf327" (UID: "04fd7ef5-cf3c-4ed7-963d-e03294eaf327"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:52.089740 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.089711    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-kube-api-access-klgts" pod "04fd7ef5-cf3c-4ed7-963d-e03294eaf327" (UID: "04fd7ef5-cf3c-4ed7-963d-e03294eaf327"). InnerVolumeSpecName "kube-api-access-klgts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:12:52.089740 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.089717    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b453729-cac0-4839-bdfa-0da04e59c1bc-istio-envoy" pod "2b453729-cac0-4839-bdfa-0da04e59c1bc" (UID: "2b453729-cac0-4839-bdfa-0da04e59c1bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:52.089912 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.089870    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b453729-cac0-4839-bdfa-0da04e59c1bc-istio-token" pod "2b453729-cac0-4839-bdfa-0da04e59c1bc" (UID: "2b453729-cac0-4839-bdfa-0da04e59c1bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:12:52.089912 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.089901    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b453729-cac0-4839-bdfa-0da04e59c1bc-kube-api-access-f55pm" pod "2b453729-cac0-4839-bdfa-0da04e59c1bc" (UID: "2b453729-cac0-4839-bdfa-0da04e59c1bc"). InnerVolumeSpecName "kube-api-access-f55pm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:12:52.090044 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.089928    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25a8498d-e23e-44a9-81fc-3170b124ad13-istio-token" pod "25a8498d-e23e-44a9-81fc-3170b124ad13" (UID: "25a8498d-e23e-44a9-81fc-3170b124ad13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:12:52.090408 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.090386    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25a8498d-e23e-44a9-81fc-3170b124ad13-kube-api-access-rp2d5" pod "25a8498d-e23e-44a9-81fc-3170b124ad13" (UID: "25a8498d-e23e-44a9-81fc-3170b124ad13"). InnerVolumeSpecName "kube-api-access-rp2d5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:12:52.090966 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.090941    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25a8498d-e23e-44a9-81fc-3170b124ad13-istio-podinfo" pod "25a8498d-e23e-44a9-81fc-3170b124ad13" (UID: "25a8498d-e23e-44a9-81fc-3170b124ad13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:12:52.091117 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.091100    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-istio-token" pod "04fd7ef5-cf3c-4ed7-963d-e03294eaf327" (UID: "04fd7ef5-cf3c-4ed7-963d-e03294eaf327"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:12:52.091225 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.091205    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-istio-podinfo" pod "04fd7ef5-cf3c-4ed7-963d-e03294eaf327" (UID: "04fd7ef5-cf3c-4ed7-963d-e03294eaf327"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:12:52.187439 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.187349    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b453729-cac0-4839-bdfa-0da04e59c1bc-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:52.187439 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.187393    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:52.187439 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.187411    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:52.187439 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.187431    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a8498d-e23e-44a9-81fc-3170b124ad13-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:52.187971 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.187450    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b453729-cac0-4839-bdfa-0da04e59c1bc-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:52.187971 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.187467    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25a8498d-e23e-44a9-81fc-3170b124ad13-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:52.187971 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.187485    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:52.187971 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.187502    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:52.187971 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.187519    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25a8498d-e23e-44a9-81fc-3170b124ad13-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:52.187971 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.187536    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25a8498d-e23e-44a9-81fc-3170b124ad13-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:52.187971 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.187552    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b453729-cac0-4839-bdfa-0da04e59c1bc-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:52.187971 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.187570    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klgts\" (UniqueName: \"kubernetes.io/projected/04fd7ef5-cf3c-4ed7-963d-e03294eaf327-kube-api-access-klgts\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:52.187971 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.187591    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25a8498d-e23e-44a9-81fc-3170b124ad13-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:52.187971 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.187660    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f55pm\" (UniqueName: \"kubernetes.io/projected/2b453729-cac0-4839-bdfa-0da04e59c1bc-kube-api-access-f55pm\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:52.187971 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.187682    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp2d5\" (UniqueName: \"kubernetes.io/projected/25a8498d-e23e-44a9-81fc-3170b124ad13-kube-api-access-rp2d5\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:52.212643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.212587    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89"]
Nov 03 18:12:52.212893 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.212879    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-sdc89"]
Nov 03 18:12:52.411036 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.411000    5315 generic.go:356] "Generic (PLEG): container finished" podID="25a8498d-e23e-44a9-81fc-3170b124ad13" containerID="0aefff3fa1372a9e3b95667273a96b7ba828e9bf2c79b586bd15234b49e7aacf" exitCode=0
Nov 03 18:12:52.411281 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.411082    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9"
Nov 03 18:12:52.411281 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.411090    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9" event={"ID":"25a8498d-e23e-44a9-81fc-3170b124ad13","Type":"ContainerDied","Data":"0aefff3fa1372a9e3b95667273a96b7ba828e9bf2c79b586bd15234b49e7aacf"}
Nov 03 18:12:52.411281 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.411179    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9" event={"ID":"25a8498d-e23e-44a9-81fc-3170b124ad13","Type":"ContainerDied","Data":"e76c9ed54e2908688ec36d5ccf481f43b623ec6da50331a6fca414c696fb549e"}
Nov 03 18:12:52.411281 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.411210    5315 scope.go:122] "RemoveContainer" containerID="0aefff3fa1372a9e3b95667273a96b7ba828e9bf2c79b586bd15234b49e7aacf"
Nov 03 18:12:52.423506 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.423469    5315 scope.go:122] "RemoveContainer" containerID="0d23428114557db8e457dc9944c3de90e0c1ea9a9527b88683aca4ec1b9a423f"
Nov 03 18:12:52.438436 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.423817    5315 generic.go:356] "Generic (PLEG): container finished" podID="04fd7ef5-cf3c-4ed7-963d-e03294eaf327" containerID="e56a7f73492bd3b33294e91ede46de4ea9ee4b02e93ea8575c5f9dbc2f12167d" exitCode=0
Nov 03 18:12:52.438436 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.423932    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82" event={"ID":"04fd7ef5-cf3c-4ed7-963d-e03294eaf327","Type":"ContainerDied","Data":"e56a7f73492bd3b33294e91ede46de4ea9ee4b02e93ea8575c5f9dbc2f12167d"}
Nov 03 18:12:52.438436 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.423988    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82" event={"ID":"04fd7ef5-cf3c-4ed7-963d-e03294eaf327","Type":"ContainerDied","Data":"f3db370de73d2f1a513cd971b77a0695d7e9fbfc7cbf62c5069e66013443e82e"}
Nov 03 18:12:52.438436 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.424081    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82"
Nov 03 18:12:52.473661 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.473574    5315 generic.go:356] "Generic (PLEG): container finished" podID="2b453729-cac0-4839-bdfa-0da04e59c1bc" containerID="0feeb0a45b8262c496c57b031f48c2254a1e7ed1a311b3614f093f5363fa1829" exitCode=0
Nov 03 18:12:52.473981 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.473954    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t" event={"ID":"2b453729-cac0-4839-bdfa-0da04e59c1bc","Type":"ContainerDied","Data":"0feeb0a45b8262c496c57b031f48c2254a1e7ed1a311b3614f093f5363fa1829"}
Nov 03 18:12:52.474173 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.474155    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t" event={"ID":"2b453729-cac0-4839-bdfa-0da04e59c1bc","Type":"ContainerDied","Data":"f6c672ba463ccdc778d718267f0191dd7d5c2a52cf3a40c2947d7d3732fd3c3b"}
Nov 03 18:12:52.474389 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.474373    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t"
Nov 03 18:12:52.480525 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.479443    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d" event={"ID":"ae6e0c1a-a26e-4b44-8e94-540944db8bd8","Type":"ContainerStarted","Data":"76fc11ded9455c2a81036b73e1425af8d992264ffeb97286ccc3e0acb07b726c"}
Nov 03 18:12:52.486696 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.486466    5315 scope.go:122] "RemoveContainer" containerID="0aefff3fa1372a9e3b95667273a96b7ba828e9bf2c79b586bd15234b49e7aacf"
Nov 03 18:12:52.487640 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:12:52.487401    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0aefff3fa1372a9e3b95667273a96b7ba828e9bf2c79b586bd15234b49e7aacf\": container with ID starting with 0aefff3fa1372a9e3b95667273a96b7ba828e9bf2c79b586bd15234b49e7aacf not found: ID does not exist" containerID="0aefff3fa1372a9e3b95667273a96b7ba828e9bf2c79b586bd15234b49e7aacf"
Nov 03 18:12:52.487640 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.487440    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aefff3fa1372a9e3b95667273a96b7ba828e9bf2c79b586bd15234b49e7aacf"} err="failed to get container status \"0aefff3fa1372a9e3b95667273a96b7ba828e9bf2c79b586bd15234b49e7aacf\": rpc error: code = NotFound desc = could not find container \"0aefff3fa1372a9e3b95667273a96b7ba828e9bf2c79b586bd15234b49e7aacf\": container with ID starting with 0aefff3fa1372a9e3b95667273a96b7ba828e9bf2c79b586bd15234b49e7aacf not found: ID does not exist"
Nov 03 18:12:52.487640 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.487463    5315 scope.go:122] "RemoveContainer" containerID="0d23428114557db8e457dc9944c3de90e0c1ea9a9527b88683aca4ec1b9a423f"
Nov 03 18:12:52.488932 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:12:52.488581    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d23428114557db8e457dc9944c3de90e0c1ea9a9527b88683aca4ec1b9a423f\": container with ID starting with 0d23428114557db8e457dc9944c3de90e0c1ea9a9527b88683aca4ec1b9a423f not found: ID does not exist" containerID="0d23428114557db8e457dc9944c3de90e0c1ea9a9527b88683aca4ec1b9a423f"
Nov 03 18:12:52.488932 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.488635    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d23428114557db8e457dc9944c3de90e0c1ea9a9527b88683aca4ec1b9a423f"} err="failed to get container status \"0d23428114557db8e457dc9944c3de90e0c1ea9a9527b88683aca4ec1b9a423f\": rpc error: code = NotFound desc = could not find container \"0d23428114557db8e457dc9944c3de90e0c1ea9a9527b88683aca4ec1b9a423f\": container with ID starting with 0d23428114557db8e457dc9944c3de90e0c1ea9a9527b88683aca4ec1b9a423f not found: ID does not exist"
Nov 03 18:12:52.488932 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.488656    5315 scope.go:122] "RemoveContainer" containerID="e56a7f73492bd3b33294e91ede46de4ea9ee4b02e93ea8575c5f9dbc2f12167d"
Nov 03 18:12:52.494345 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.494321    5315 generic.go:356] "Generic (PLEG): container finished" podID="375f4e00-8c81-4de5-97bf-9be54ea028ed" containerID="79090806e6131c0c136e68a076d2e9ad0df849ff4976a5d41b06c0155b0be5f2" exitCode=0
Nov 03 18:12:52.494522 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.494501    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm" event={"ID":"375f4e00-8c81-4de5-97bf-9be54ea028ed","Type":"ContainerDied","Data":"79090806e6131c0c136e68a076d2e9ad0df849ff4976a5d41b06c0155b0be5f2"}
Nov 03 18:12:52.494684 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.494666    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm" event={"ID":"375f4e00-8c81-4de5-97bf-9be54ea028ed","Type":"ContainerDied","Data":"814bb639cb36888ce0cae90a0cb4115fb7a1ce49c1dc267297e540d19f293acf"}
Nov 03 18:12:52.495041 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.495018    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm"
Nov 03 18:12:52.580270 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.578741    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb"]
Nov 03 18:12:52.587555 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.585091    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wt5mb"]
Nov 03 18:12:52.587555 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.587227    5315 scope.go:122] "RemoveContainer" containerID="e56a7f73492bd3b33294e91ede46de4ea9ee4b02e93ea8575c5f9dbc2f12167d"
Nov 03 18:12:52.587876 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:12:52.587672    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e56a7f73492bd3b33294e91ede46de4ea9ee4b02e93ea8575c5f9dbc2f12167d\": container with ID starting with e56a7f73492bd3b33294e91ede46de4ea9ee4b02e93ea8575c5f9dbc2f12167d not found: ID does not exist" containerID="e56a7f73492bd3b33294e91ede46de4ea9ee4b02e93ea8575c5f9dbc2f12167d"
Nov 03 18:12:52.587876 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.587709    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e56a7f73492bd3b33294e91ede46de4ea9ee4b02e93ea8575c5f9dbc2f12167d"} err="failed to get container status \"e56a7f73492bd3b33294e91ede46de4ea9ee4b02e93ea8575c5f9dbc2f12167d\": rpc error: code = NotFound desc = could not find container \"e56a7f73492bd3b33294e91ede46de4ea9ee4b02e93ea8575c5f9dbc2f12167d\": container with ID starting with e56a7f73492bd3b33294e91ede46de4ea9ee4b02e93ea8575c5f9dbc2f12167d not found: ID does not exist"
Nov 03 18:12:52.587876 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.587732    5315 scope.go:122] "RemoveContainer" containerID="0feeb0a45b8262c496c57b031f48c2254a1e7ed1a311b3614f093f5363fa1829"
Nov 03 18:12:52.602173 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.602055    5315 scope.go:122] "RemoveContainer" containerID="0feeb0a45b8262c496c57b031f48c2254a1e7ed1a311b3614f093f5363fa1829"
Nov 03 18:12:52.602455 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:12:52.602427    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0feeb0a45b8262c496c57b031f48c2254a1e7ed1a311b3614f093f5363fa1829\": container with ID starting with 0feeb0a45b8262c496c57b031f48c2254a1e7ed1a311b3614f093f5363fa1829 not found: ID does not exist" containerID="0feeb0a45b8262c496c57b031f48c2254a1e7ed1a311b3614f093f5363fa1829"
Nov 03 18:12:52.602552 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.602468    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0feeb0a45b8262c496c57b031f48c2254a1e7ed1a311b3614f093f5363fa1829"} err="failed to get container status \"0feeb0a45b8262c496c57b031f48c2254a1e7ed1a311b3614f093f5363fa1829\": rpc error: code = NotFound desc = could not find container \"0feeb0a45b8262c496c57b031f48c2254a1e7ed1a311b3614f093f5363fa1829\": container with ID starting with 0feeb0a45b8262c496c57b031f48c2254a1e7ed1a311b3614f093f5363fa1829 not found: ID does not exist"
Nov 03 18:12:52.602552 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.602494    5315 scope.go:122] "RemoveContainer" containerID="79090806e6131c0c136e68a076d2e9ad0df849ff4976a5d41b06c0155b0be5f2"
Nov 03 18:12:52.613120 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.613101    5315 scope.go:122] "RemoveContainer" containerID="79090806e6131c0c136e68a076d2e9ad0df849ff4976a5d41b06c0155b0be5f2"
Nov 03 18:12:52.613398 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:12:52.613381    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79090806e6131c0c136e68a076d2e9ad0df849ff4976a5d41b06c0155b0be5f2\": container with ID starting with 79090806e6131c0c136e68a076d2e9ad0df849ff4976a5d41b06c0155b0be5f2 not found: ID does not exist" containerID="79090806e6131c0c136e68a076d2e9ad0df849ff4976a5d41b06c0155b0be5f2"
Nov 03 18:12:52.613445 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.613405    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79090806e6131c0c136e68a076d2e9ad0df849ff4976a5d41b06c0155b0be5f2"} err="failed to get container status \"79090806e6131c0c136e68a076d2e9ad0df849ff4976a5d41b06c0155b0be5f2\": rpc error: code = NotFound desc = could not find container \"79090806e6131c0c136e68a076d2e9ad0df849ff4976a5d41b06c0155b0be5f2\": container with ID starting with 79090806e6131c0c136e68a076d2e9ad0df849ff4976a5d41b06c0155b0be5f2 not found: ID does not exist"
Nov 03 18:12:52.763757 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.763658    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d"
Nov 03 18:12:52.764661 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.764601    5315 patch_prober.go:29] interesting pod/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 03 18:12:52.764754 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.764698    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d" podUID="ae6e0c1a-a26e-4b44-8e94-540944db8bd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 03 18:12:52.918917 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.918885    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p"]
Nov 03 18:12:52.922910 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:52.922884    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-lqr5p"]
Nov 03 18:12:53.017340 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:53.017253    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9"]
Nov 03 18:12:53.021048 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:53.021020    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-55nc9"]
Nov 03 18:12:53.107255 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:53.107221    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e4ca152-3632-485f-94b2-4f430cbdfe4c" path="/var/lib/kubelet/pods/1e4ca152-3632-485f-94b2-4f430cbdfe4c/volumes"
Nov 03 18:12:53.107773 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:53.107759    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25a8498d-e23e-44a9-81fc-3170b124ad13" path="/var/lib/kubelet/pods/25a8498d-e23e-44a9-81fc-3170b124ad13/volumes"
Nov 03 18:12:53.108217 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:53.108204    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57d6511c-f9f2-4ca7-9bfb-6475dd17f94b" path="/var/lib/kubelet/pods/57d6511c-f9f2-4ca7-9bfb-6475dd17f94b/volumes"
Nov 03 18:12:53.108990 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:53.108974    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79050b41-0f3b-4fc4-a8e8-50316359ac33" path="/var/lib/kubelet/pods/79050b41-0f3b-4fc4-a8e8-50316359ac33/volumes"
Nov 03 18:12:53.109420 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:53.109407    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="858e4f16-d2d2-4ba6-9e75-b81fe8ad5674" path="/var/lib/kubelet/pods/858e4f16-d2d2-4ba6-9e75-b81fe8ad5674/volumes"
Nov 03 18:12:53.122508 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:53.122482    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t"]
Nov 03 18:12:53.154029 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:53.153992    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-qzc8t"]
Nov 03 18:12:53.253427 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:53.253388    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm"]
Nov 03 18:12:53.256905 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:53.256868    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-dxmbm"]
Nov 03 18:12:53.460581 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:53.460544    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82"]
Nov 03 18:12:53.464552 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:53.464506    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-95d82"]
Nov 03 18:12:53.493317 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:53.493279    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d"
Nov 03 18:12:53.764040 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:53.763958    5315 patch_prober.go:29] interesting pod/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 03 18:12:53.764040 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:53.764016    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d" podUID="ae6e0c1a-a26e-4b44-8e94-540944db8bd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 03 18:12:54.578753 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:12:54.578713    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:12:54.763937 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:54.763901    5315 patch_prober.go:29] interesting pod/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 03 18:12:54.764129 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:54.763977    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d" podUID="ae6e0c1a-a26e-4b44-8e94-540944db8bd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 03 18:12:55.110822 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:55.110785    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04fd7ef5-cf3c-4ed7-963d-e03294eaf327" path="/var/lib/kubelet/pods/04fd7ef5-cf3c-4ed7-963d-e03294eaf327/volumes"
Nov 03 18:12:55.111374 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:55.111359    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b453729-cac0-4839-bdfa-0da04e59c1bc" path="/var/lib/kubelet/pods/2b453729-cac0-4839-bdfa-0da04e59c1bc/volumes"
Nov 03 18:12:55.111831 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:55.111818    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="375f4e00-8c81-4de5-97bf-9be54ea028ed" path="/var/lib/kubelet/pods/375f4e00-8c81-4de5-97bf-9be54ea028ed/volumes"
Nov 03 18:12:55.763727 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:55.763692    5315 patch_prober.go:29] interesting pod/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 03 18:12:55.764249 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:55.763751    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d" podUID="ae6e0c1a-a26e-4b44-8e94-540944db8bd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 03 18:12:56.763792 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:56.763753    5315 patch_prober.go:29] interesting pod/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 03 18:12:56.764310 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:56.763840    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d" podUID="ae6e0c1a-a26e-4b44-8e94-540944db8bd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 03 18:12:57.764141 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:57.764102    5315 patch_prober.go:29] interesting pod/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 03 18:12:57.764693 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:57.764166    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d" podUID="ae6e0c1a-a26e-4b44-8e94-540944db8bd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 03 18:12:58.768797 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:58.768765    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d"
Nov 03 18:12:58.770079 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:12:58.770055    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d"
Nov 03 18:13:01.833792 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:01.833746    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45"]
Nov 03 18:13:01.834341 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:01.834046    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45" podUID="4e8997a5-bfb9-4517-bb5e-632f8639e44d" containerName="istio-proxy" containerID="cri-o://2975927156f1c80407a69ee06540379ec4bdcc7d0e458453e8d773bdaaa70515" gracePeriod=30
Nov 03 18:13:01.880127 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:01.880083    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp"]
Nov 03 18:13:01.880350 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:01.880305    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp" podUID="59f3868f-6b95-4d02-afe7-4e5228cf37c3" containerName="istio-proxy" containerID="cri-o://e9f733f4fd8b9f3f646c0164b8eb5616c00eca87ff3c03c464c9217d024cc71f" gracePeriod=30
Nov 03 18:13:01.884767 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:01.884723    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj"]
Nov 03 18:13:01.885032 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:01.884984    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj" podUID="bd9b0713-a9c7-4344-8b32-487d235f9a72" containerName="istio-proxy" containerID="cri-o://043eaf103f43bb3b1032c7ba51943da1b159dbd12fd55c4ce69d1f58a2998d23" gracePeriod=30
Nov 03 18:13:01.926979 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:01.926943    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff"]
Nov 03 18:13:01.927173 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:01.927143    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff" podUID="ad113518-40b2-4d26-b886-a7f99afdf6f8" containerName="istio-proxy" containerID="cri-o://e6beb19892d462933e2480386ca6c9fb41d3fb186fa0a6e2879c86a73ba7a53d" gracePeriod=30
Nov 03 18:13:04.135474 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:04.135442    5315 scope.go:122] "RemoveContainer" containerID="d04c61463bf9e0719fddcc2c45b1b15db890bd2c76e95be9d404456c268fc6dc"
Nov 03 18:13:04.145965 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:04.145935    5315 scope.go:122] "RemoveContainer" containerID="c4f3e8791f7695bc87d8be1a5bc9933b2c479777778756cd32e002233f468dcd"
Nov 03 18:13:04.155857 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:04.155834    5315 scope.go:122] "RemoveContainer" containerID="68668f9bf2d9483d82f7396878b4345af80cf2bffc1a140ef85e1a69103abfc8"
Nov 03 18:13:04.165604 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:04.165578    5315 scope.go:122] "RemoveContainer" containerID="becc088e90162c524ca741adf4135fcbe7db9021e2dadb79e3ff33aa8f9d20f0"
Nov 03 18:13:04.175172 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:04.175149    5315 scope.go:122] "RemoveContainer" containerID="67915755c38e19d2b6c61f43435a204693c61a24d80eb679495b98d3ef8649cb"
Nov 03 18:13:04.184388 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:04.184369    5315 scope.go:122] "RemoveContainer" containerID="0371632aa5e4fefeb7113787793526666eef8ed3cc33936c5a39c5d52c4754b5"
Nov 03 18:13:04.194097 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:04.194070    5315 scope.go:122] "RemoveContainer" containerID="ef7b79738d942825e18db8e65849fea2d35213b745c0876ef7ddbae73baede31"
Nov 03 18:13:04.204101 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:04.204080    5315 scope.go:122] "RemoveContainer" containerID="435ed6f4608146f2941f806a756f18a9a6b48e00b313ae574c631f858d7d657d"
Nov 03 18:13:04.213947 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:04.213924    5315 scope.go:122] "RemoveContainer" containerID="b71ac0ea402dbd2ed532dc68a2ede773659b36a65cf39c7f70cd9bded6c60d00"
Nov 03 18:13:04.223786 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:04.223762    5315 scope.go:122] "RemoveContainer" containerID="314ef8a11b70c4fb76504000b9c052bb100bd7709a84fb89d6b905e591643b91"
Nov 03 18:13:04.233328 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:04.233308    5315 scope.go:122] "RemoveContainer" containerID="33401f9d2ff4ef361cef300206bff2787c0ba7184b7db672c584edf9f2397b71"
Nov 03 18:13:04.243218 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:04.243195    5315 scope.go:122] "RemoveContainer" containerID="af04986322ab502bfde031edaab5efacd4b5f367d4223c7dabaa06654be3c6ba"
Nov 03 18:13:04.252481 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:04.252461    5315 scope.go:122] "RemoveContainer" containerID="a1f7c973f11c0bca3a70ceacf0e1dc17ea76b666dd2cca98f3a95f25ec28a551"
Nov 03 18:13:06.306740 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:06.306707    5315 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:13:06.307134 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:06.306762    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff" podUID="ad113518-40b2-4d26-b886-a7f99afdf6f8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:13:07.274847 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.274456    5315 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 03 18:13:07.274847 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.274513    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45" podUID="4e8997a5-bfb9-4517-bb5e-632f8639e44d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 03 18:13:07.389449 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.389422    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp"
Nov 03 18:13:07.432444 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.432417    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff"
Nov 03 18:13:07.488935 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.488825    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj"
Nov 03 18:13:07.490499 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.490470    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59f3868f-6b95-4d02-afe7-4e5228cf37c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59f3868f-6b95-4d02-afe7-4e5228cf37c3-istio-podinfo\") pod \"59f3868f-6b95-4d02-afe7-4e5228cf37c3\" (UID: \"59f3868f-6b95-4d02-afe7-4e5228cf37c3\") "
Nov 03 18:13:07.490663 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.490592    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59f3868f-6b95-4d02-afe7-4e5228cf37c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f3868f-6b95-4d02-afe7-4e5228cf37c3-credential-socket\") pod \"59f3868f-6b95-4d02-afe7-4e5228cf37c3\" (UID: \"59f3868f-6b95-4d02-afe7-4e5228cf37c3\") "
Nov 03 18:13:07.490747 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.490715    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59f3868f-6b95-4d02-afe7-4e5228cf37c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59f3868f-6b95-4d02-afe7-4e5228cf37c3-istio-envoy\") pod \"59f3868f-6b95-4d02-afe7-4e5228cf37c3\" (UID: \"59f3868f-6b95-4d02-afe7-4e5228cf37c3\") "
Nov 03 18:13:07.490747 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.490745    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59f3868f-6b95-4d02-afe7-4e5228cf37c3-kube-api-access-lpbd9\" (UniqueName: \"kubernetes.io/projected/59f3868f-6b95-4d02-afe7-4e5228cf37c3-kube-api-access-lpbd9\") pod \"59f3868f-6b95-4d02-afe7-4e5228cf37c3\" (UID: \"59f3868f-6b95-4d02-afe7-4e5228cf37c3\") "
Nov 03 18:13:07.490882 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.490866    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59f3868f-6b95-4d02-afe7-4e5228cf37c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59f3868f-6b95-4d02-afe7-4e5228cf37c3-istio-data\") pod \"59f3868f-6b95-4d02-afe7-4e5228cf37c3\" (UID: \"59f3868f-6b95-4d02-afe7-4e5228cf37c3\") "
Nov 03 18:13:07.490948 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.490905    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59f3868f-6b95-4d02-afe7-4e5228cf37c3-istio-token\" (UniqueName: \"kubernetes.io/projected/59f3868f-6b95-4d02-afe7-4e5228cf37c3-istio-token\") pod \"59f3868f-6b95-4d02-afe7-4e5228cf37c3\" (UID: \"59f3868f-6b95-4d02-afe7-4e5228cf37c3\") "
Nov 03 18:13:07.491048 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.491032    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59f3868f-6b95-4d02-afe7-4e5228cf37c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59f3868f-6b95-4d02-afe7-4e5228cf37c3-workload-certs\") pod \"59f3868f-6b95-4d02-afe7-4e5228cf37c3\" (UID: \"59f3868f-6b95-4d02-afe7-4e5228cf37c3\") "
Nov 03 18:13:07.491115 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.491100    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59f3868f-6b95-4d02-afe7-4e5228cf37c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59f3868f-6b95-4d02-afe7-4e5228cf37c3-istiod-ca-cert\") pod \"59f3868f-6b95-4d02-afe7-4e5228cf37c3\" (UID: \"59f3868f-6b95-4d02-afe7-4e5228cf37c3\") "
Nov 03 18:13:07.491171 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.491159    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59f3868f-6b95-4d02-afe7-4e5228cf37c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f3868f-6b95-4d02-afe7-4e5228cf37c3-workload-socket\") pod \"59f3868f-6b95-4d02-afe7-4e5228cf37c3\" (UID: \"59f3868f-6b95-4d02-afe7-4e5228cf37c3\") "
Nov 03 18:13:07.491519 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.491494    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f3868f-6b95-4d02-afe7-4e5228cf37c3-credential-socket" pod "59f3868f-6b95-4d02-afe7-4e5228cf37c3" (UID: "59f3868f-6b95-4d02-afe7-4e5228cf37c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:13:07.492196 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.491937    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f3868f-6b95-4d02-afe7-4e5228cf37c3-workload-socket" pod "59f3868f-6b95-4d02-afe7-4e5228cf37c3" (UID: "59f3868f-6b95-4d02-afe7-4e5228cf37c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:13:07.493478 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.492789    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f3868f-6b95-4d02-afe7-4e5228cf37c3-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:07.493718 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.493687    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f3868f-6b95-4d02-afe7-4e5228cf37c3-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:07.493835 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.493075    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f3868f-6b95-4d02-afe7-4e5228cf37c3-istio-data" pod "59f3868f-6b95-4d02-afe7-4e5228cf37c3" (UID: "59f3868f-6b95-4d02-afe7-4e5228cf37c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:13:07.494388 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.494367    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59f3868f-6b95-4d02-afe7-4e5228cf37c3-istiod-ca-cert" pod "59f3868f-6b95-4d02-afe7-4e5228cf37c3" (UID: "59f3868f-6b95-4d02-afe7-4e5228cf37c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:13:07.495988 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.495909    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f3868f-6b95-4d02-afe7-4e5228cf37c3-workload-certs" pod "59f3868f-6b95-4d02-afe7-4e5228cf37c3" (UID: "59f3868f-6b95-4d02-afe7-4e5228cf37c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:13:07.497445 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.497407    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59f3868f-6b95-4d02-afe7-4e5228cf37c3-kube-api-access-lpbd9" pod "59f3868f-6b95-4d02-afe7-4e5228cf37c3" (UID: "59f3868f-6b95-4d02-afe7-4e5228cf37c3"). InnerVolumeSpecName "kube-api-access-lpbd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:13:07.499242 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.499208    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f3868f-6b95-4d02-afe7-4e5228cf37c3-istio-envoy" pod "59f3868f-6b95-4d02-afe7-4e5228cf37c3" (UID: "59f3868f-6b95-4d02-afe7-4e5228cf37c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:13:07.500751 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.500726    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59f3868f-6b95-4d02-afe7-4e5228cf37c3-istio-podinfo" pod "59f3868f-6b95-4d02-afe7-4e5228cf37c3" (UID: "59f3868f-6b95-4d02-afe7-4e5228cf37c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:13:07.501203 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.501164    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59f3868f-6b95-4d02-afe7-4e5228cf37c3-istio-token" pod "59f3868f-6b95-4d02-afe7-4e5228cf37c3" (UID: "59f3868f-6b95-4d02-afe7-4e5228cf37c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:13:07.506747 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.506724    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45"
Nov 03 18:13:07.608123 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.601361    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd9b0713-a9c7-4344-8b32-487d235f9a72-istio-token\" (UniqueName: \"kubernetes.io/projected/bd9b0713-a9c7-4344-8b32-487d235f9a72-istio-token\") pod \"bd9b0713-a9c7-4344-8b32-487d235f9a72\" (UID: \"bd9b0713-a9c7-4344-8b32-487d235f9a72\") "
Nov 03 18:13:07.608123 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.601438    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd9b0713-a9c7-4344-8b32-487d235f9a72-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd9b0713-a9c7-4344-8b32-487d235f9a72-workload-certs\") pod \"bd9b0713-a9c7-4344-8b32-487d235f9a72\" (UID: \"bd9b0713-a9c7-4344-8b32-487d235f9a72\") "
Nov 03 18:13:07.608123 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.601473    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad113518-40b2-4d26-b886-a7f99afdf6f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad113518-40b2-4d26-b886-a7f99afdf6f8-workload-socket\") pod \"ad113518-40b2-4d26-b886-a7f99afdf6f8\" (UID: \"ad113518-40b2-4d26-b886-a7f99afdf6f8\") "
Nov 03 18:13:07.608123 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.601516    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e8997a5-bfb9-4517-bb5e-632f8639e44d-istio-token\" (UniqueName: \"kubernetes.io/projected/4e8997a5-bfb9-4517-bb5e-632f8639e44d-istio-token\") pod \"4e8997a5-bfb9-4517-bb5e-632f8639e44d\" (UID: \"4e8997a5-bfb9-4517-bb5e-632f8639e44d\") "
Nov 03 18:13:07.610003 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.601599    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd9b0713-a9c7-4344-8b32-487d235f9a72-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd9b0713-a9c7-4344-8b32-487d235f9a72-istio-podinfo\") pod \"bd9b0713-a9c7-4344-8b32-487d235f9a72\" (UID: \"bd9b0713-a9c7-4344-8b32-487d235f9a72\") "
Nov 03 18:13:07.610003 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.608587    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad113518-40b2-4d26-b886-a7f99afdf6f8-kube-api-access-4z8ld\" (UniqueName: \"kubernetes.io/projected/ad113518-40b2-4d26-b886-a7f99afdf6f8-kube-api-access-4z8ld\") pod \"ad113518-40b2-4d26-b886-a7f99afdf6f8\" (UID: \"ad113518-40b2-4d26-b886-a7f99afdf6f8\") "
Nov 03 18:13:07.610003 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.608690    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd9b0713-a9c7-4344-8b32-487d235f9a72-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd9b0713-a9c7-4344-8b32-487d235f9a72-credential-socket\") pod \"bd9b0713-a9c7-4344-8b32-487d235f9a72\" (UID: \"bd9b0713-a9c7-4344-8b32-487d235f9a72\") "
Nov 03 18:13:07.610003 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.608727    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e8997a5-bfb9-4517-bb5e-632f8639e44d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e8997a5-bfb9-4517-bb5e-632f8639e44d-istiod-ca-cert\") pod \"4e8997a5-bfb9-4517-bb5e-632f8639e44d\" (UID: \"4e8997a5-bfb9-4517-bb5e-632f8639e44d\") "
Nov 03 18:13:07.610003 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.608758    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad113518-40b2-4d26-b886-a7f99afdf6f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad113518-40b2-4d26-b886-a7f99afdf6f8-istiod-ca-cert\") pod \"ad113518-40b2-4d26-b886-a7f99afdf6f8\" (UID: \"ad113518-40b2-4d26-b886-a7f99afdf6f8\") "
Nov 03 18:13:07.610003 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.608798    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e8997a5-bfb9-4517-bb5e-632f8639e44d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8997a5-bfb9-4517-bb5e-632f8639e44d-credential-socket\") pod \"4e8997a5-bfb9-4517-bb5e-632f8639e44d\" (UID: \"4e8997a5-bfb9-4517-bb5e-632f8639e44d\") "
Nov 03 18:13:07.610003 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.608837    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e8997a5-bfb9-4517-bb5e-632f8639e44d-kube-api-access-whghb\" (UniqueName: \"kubernetes.io/projected/4e8997a5-bfb9-4517-bb5e-632f8639e44d-kube-api-access-whghb\") pod \"4e8997a5-bfb9-4517-bb5e-632f8639e44d\" (UID: \"4e8997a5-bfb9-4517-bb5e-632f8639e44d\") "
Nov 03 18:13:07.610003 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.608911    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e8997a5-bfb9-4517-bb5e-632f8639e44d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e8997a5-bfb9-4517-bb5e-632f8639e44d-istio-data\") pod \"4e8997a5-bfb9-4517-bb5e-632f8639e44d\" (UID: \"4e8997a5-bfb9-4517-bb5e-632f8639e44d\") "
Nov 03 18:13:07.610003 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.608960    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e8997a5-bfb9-4517-bb5e-632f8639e44d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e8997a5-bfb9-4517-bb5e-632f8639e44d-workload-certs\") pod \"4e8997a5-bfb9-4517-bb5e-632f8639e44d\" (UID: \"4e8997a5-bfb9-4517-bb5e-632f8639e44d\") "
Nov 03 18:13:07.610003 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.608994    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd9b0713-a9c7-4344-8b32-487d235f9a72-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd9b0713-a9c7-4344-8b32-487d235f9a72-workload-socket\") pod \"bd9b0713-a9c7-4344-8b32-487d235f9a72\" (UID: \"bd9b0713-a9c7-4344-8b32-487d235f9a72\") "
Nov 03 18:13:07.610003 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.609031    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad113518-40b2-4d26-b886-a7f99afdf6f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad113518-40b2-4d26-b886-a7f99afdf6f8-credential-socket\") pod \"ad113518-40b2-4d26-b886-a7f99afdf6f8\" (UID: \"ad113518-40b2-4d26-b886-a7f99afdf6f8\") "
Nov 03 18:13:07.610003 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.609080    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad113518-40b2-4d26-b886-a7f99afdf6f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad113518-40b2-4d26-b886-a7f99afdf6f8-istio-podinfo\") pod \"ad113518-40b2-4d26-b886-a7f99afdf6f8\" (UID: \"ad113518-40b2-4d26-b886-a7f99afdf6f8\") "
Nov 03 18:13:07.610003 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.609138    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e8997a5-bfb9-4517-bb5e-632f8639e44d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e8997a5-bfb9-4517-bb5e-632f8639e44d-istio-podinfo\") pod \"4e8997a5-bfb9-4517-bb5e-632f8639e44d\" (UID: \"4e8997a5-bfb9-4517-bb5e-632f8639e44d\") "
Nov 03 18:13:07.610003 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.609165    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e8997a5-bfb9-4517-bb5e-632f8639e44d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e8997a5-bfb9-4517-bb5e-632f8639e44d-istio-envoy\") pod \"4e8997a5-bfb9-4517-bb5e-632f8639e44d\" (UID: \"4e8997a5-bfb9-4517-bb5e-632f8639e44d\") "
Nov 03 18:13:07.610003 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.609199    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd9b0713-a9c7-4344-8b32-487d235f9a72-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd9b0713-a9c7-4344-8b32-487d235f9a72-istio-envoy\") pod \"bd9b0713-a9c7-4344-8b32-487d235f9a72\" (UID: \"bd9b0713-a9c7-4344-8b32-487d235f9a72\") "
Nov 03 18:13:07.610003 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.609325    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd9b0713-a9c7-4344-8b32-487d235f9a72-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd9b0713-a9c7-4344-8b32-487d235f9a72-istio-data\") pod \"bd9b0713-a9c7-4344-8b32-487d235f9a72\" (UID: \"bd9b0713-a9c7-4344-8b32-487d235f9a72\") "
Nov 03 18:13:07.610003 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.609404    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd9b0713-a9c7-4344-8b32-487d235f9a72-kube-api-access-9cnqk\" (UniqueName: \"kubernetes.io/projected/bd9b0713-a9c7-4344-8b32-487d235f9a72-kube-api-access-9cnqk\") pod \"bd9b0713-a9c7-4344-8b32-487d235f9a72\" (UID: \"bd9b0713-a9c7-4344-8b32-487d235f9a72\") "
Nov 03 18:13:07.610003 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.609436    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad113518-40b2-4d26-b886-a7f99afdf6f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad113518-40b2-4d26-b886-a7f99afdf6f8-istio-envoy\") pod \"ad113518-40b2-4d26-b886-a7f99afdf6f8\" (UID: \"ad113518-40b2-4d26-b886-a7f99afdf6f8\") "
Nov 03 18:13:07.610003 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.609492    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad113518-40b2-4d26-b886-a7f99afdf6f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad113518-40b2-4d26-b886-a7f99afdf6f8-istio-data\") pod \"ad113518-40b2-4d26-b886-a7f99afdf6f8\" (UID: \"ad113518-40b2-4d26-b886-a7f99afdf6f8\") "
Nov 03 18:13:07.610003 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.609544    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd9b0713-a9c7-4344-8b32-487d235f9a72-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd9b0713-a9c7-4344-8b32-487d235f9a72-istiod-ca-cert\") pod \"bd9b0713-a9c7-4344-8b32-487d235f9a72\" (UID: \"bd9b0713-a9c7-4344-8b32-487d235f9a72\") "
Nov 03 18:13:07.612109 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.611678    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad113518-40b2-4d26-b886-a7f99afdf6f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad113518-40b2-4d26-b886-a7f99afdf6f8-workload-certs\") pod \"ad113518-40b2-4d26-b886-a7f99afdf6f8\" (UID: \"ad113518-40b2-4d26-b886-a7f99afdf6f8\") "
Nov 03 18:13:07.612109 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.611873    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e8997a5-bfb9-4517-bb5e-632f8639e44d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8997a5-bfb9-4517-bb5e-632f8639e44d-workload-socket\") pod \"4e8997a5-bfb9-4517-bb5e-632f8639e44d\" (UID: \"4e8997a5-bfb9-4517-bb5e-632f8639e44d\") "
Nov 03 18:13:07.612109 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.612033    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad113518-40b2-4d26-b886-a7f99afdf6f8-istio-token\" (UniqueName: \"kubernetes.io/projected/ad113518-40b2-4d26-b886-a7f99afdf6f8-istio-token\") pod \"ad113518-40b2-4d26-b886-a7f99afdf6f8\" (UID: \"ad113518-40b2-4d26-b886-a7f99afdf6f8\") "
Nov 03 18:13:07.614525 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.613722    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59f3868f-6b95-4d02-afe7-4e5228cf37c3-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:07.614525 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.613752    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59f3868f-6b95-4d02-afe7-4e5228cf37c3-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:07.614525 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.613769    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59f3868f-6b95-4d02-afe7-4e5228cf37c3-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:07.614525 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.613782    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59f3868f-6b95-4d02-afe7-4e5228cf37c3-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:07.614525 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.613797    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpbd9\" (UniqueName: \"kubernetes.io/projected/59f3868f-6b95-4d02-afe7-4e5228cf37c3-kube-api-access-lpbd9\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:07.614525 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.613812    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59f3868f-6b95-4d02-afe7-4e5228cf37c3-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:07.614525 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.613825    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59f3868f-6b95-4d02-afe7-4e5228cf37c3-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:07.616249 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.616222    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e8997a5-bfb9-4517-bb5e-632f8639e44d-workload-certs" pod "4e8997a5-bfb9-4517-bb5e-632f8639e44d" (UID: "4e8997a5-bfb9-4517-bb5e-632f8639e44d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:13:07.616911 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.616888    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd9b0713-a9c7-4344-8b32-487d235f9a72-credential-socket" pod "bd9b0713-a9c7-4344-8b32-487d235f9a72" (UID: "bd9b0713-a9c7-4344-8b32-487d235f9a72"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:13:07.617417 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.617395    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e8997a5-bfb9-4517-bb5e-632f8639e44d-istiod-ca-cert" pod "4e8997a5-bfb9-4517-bb5e-632f8639e44d" (UID: "4e8997a5-bfb9-4517-bb5e-632f8639e44d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:13:07.621810 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.621782    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad113518-40b2-4d26-b886-a7f99afdf6f8-istiod-ca-cert" pod "ad113518-40b2-4d26-b886-a7f99afdf6f8" (UID: "ad113518-40b2-4d26-b886-a7f99afdf6f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:13:07.623382 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.623355    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e8997a5-bfb9-4517-bb5e-632f8639e44d-credential-socket" pod "4e8997a5-bfb9-4517-bb5e-632f8639e44d" (UID: "4e8997a5-bfb9-4517-bb5e-632f8639e44d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:13:07.623555 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.623529    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad113518-40b2-4d26-b886-a7f99afdf6f8-workload-socket" pod "ad113518-40b2-4d26-b886-a7f99afdf6f8" (UID: "ad113518-40b2-4d26-b886-a7f99afdf6f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:13:07.623902 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.623880    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e8997a5-bfb9-4517-bb5e-632f8639e44d-istio-data" pod "4e8997a5-bfb9-4517-bb5e-632f8639e44d" (UID: "4e8997a5-bfb9-4517-bb5e-632f8639e44d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:13:07.624263 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.624242    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd9b0713-a9c7-4344-8b32-487d235f9a72-workload-certs" pod "bd9b0713-a9c7-4344-8b32-487d235f9a72" (UID: "bd9b0713-a9c7-4344-8b32-487d235f9a72"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:13:07.641995 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.641909    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd9b0713-a9c7-4344-8b32-487d235f9a72-istio-data" pod "bd9b0713-a9c7-4344-8b32-487d235f9a72" (UID: "bd9b0713-a9c7-4344-8b32-487d235f9a72"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:13:07.642581 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.642538    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd9b0713-a9c7-4344-8b32-487d235f9a72-workload-socket" pod "bd9b0713-a9c7-4344-8b32-487d235f9a72" (UID: "bd9b0713-a9c7-4344-8b32-487d235f9a72"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:13:07.642992 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.642971    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad113518-40b2-4d26-b886-a7f99afdf6f8-credential-socket" pod "ad113518-40b2-4d26-b886-a7f99afdf6f8" (UID: "ad113518-40b2-4d26-b886-a7f99afdf6f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:13:07.657833 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.657561    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad113518-40b2-4d26-b886-a7f99afdf6f8-istio-data" pod "ad113518-40b2-4d26-b886-a7f99afdf6f8" (UID: "ad113518-40b2-4d26-b886-a7f99afdf6f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:13:07.658459 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.658194    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd9b0713-a9c7-4344-8b32-487d235f9a72-istiod-ca-cert" pod "bd9b0713-a9c7-4344-8b32-487d235f9a72" (UID: "bd9b0713-a9c7-4344-8b32-487d235f9a72"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:13:07.658865 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.658657    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad113518-40b2-4d26-b886-a7f99afdf6f8-workload-certs" pod "ad113518-40b2-4d26-b886-a7f99afdf6f8" (UID: "ad113518-40b2-4d26-b886-a7f99afdf6f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:13:07.659085 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.659051    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e8997a5-bfb9-4517-bb5e-632f8639e44d-workload-socket" pod "4e8997a5-bfb9-4517-bb5e-632f8639e44d" (UID: "4e8997a5-bfb9-4517-bb5e-632f8639e44d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:13:07.663102 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.662640    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd9b0713-a9c7-4344-8b32-487d235f9a72-istio-podinfo" pod "bd9b0713-a9c7-4344-8b32-487d235f9a72" (UID: "bd9b0713-a9c7-4344-8b32-487d235f9a72"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:13:07.666198 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.666146    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e8997a5-bfb9-4517-bb5e-632f8639e44d-istio-envoy" pod "4e8997a5-bfb9-4517-bb5e-632f8639e44d" (UID: "4e8997a5-bfb9-4517-bb5e-632f8639e44d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:13:07.666835 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.666743    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd9b0713-a9c7-4344-8b32-487d235f9a72-kube-api-access-9cnqk" pod "bd9b0713-a9c7-4344-8b32-487d235f9a72" (UID: "bd9b0713-a9c7-4344-8b32-487d235f9a72"). InnerVolumeSpecName "kube-api-access-9cnqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:13:07.667563 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.667542    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad113518-40b2-4d26-b886-a7f99afdf6f8-istio-envoy" pod "ad113518-40b2-4d26-b886-a7f99afdf6f8" (UID: "ad113518-40b2-4d26-b886-a7f99afdf6f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:13:07.669871 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.669836    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e8997a5-bfb9-4517-bb5e-632f8639e44d-istio-podinfo" pod "4e8997a5-bfb9-4517-bb5e-632f8639e44d" (UID: "4e8997a5-bfb9-4517-bb5e-632f8639e44d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:13:07.672241 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.672170    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd9b0713-a9c7-4344-8b32-487d235f9a72-istio-envoy" pod "bd9b0713-a9c7-4344-8b32-487d235f9a72" (UID: "bd9b0713-a9c7-4344-8b32-487d235f9a72"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:13:07.710630 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.703514    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad113518-40b2-4d26-b886-a7f99afdf6f8-istio-token" pod "ad113518-40b2-4d26-b886-a7f99afdf6f8" (UID: "ad113518-40b2-4d26-b886-a7f99afdf6f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:13:07.710630 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.704482    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e8997a5-bfb9-4517-bb5e-632f8639e44d-kube-api-access-whghb" pod "4e8997a5-bfb9-4517-bb5e-632f8639e44d" (UID: "4e8997a5-bfb9-4517-bb5e-632f8639e44d"). InnerVolumeSpecName "kube-api-access-whghb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:13:07.710630 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.704573    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e8997a5-bfb9-4517-bb5e-632f8639e44d-istio-token" pod "4e8997a5-bfb9-4517-bb5e-632f8639e44d" (UID: "4e8997a5-bfb9-4517-bb5e-632f8639e44d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:13:07.710630 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.708690    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad113518-40b2-4d26-b886-a7f99afdf6f8-kube-api-access-4z8ld" pod "ad113518-40b2-4d26-b886-a7f99afdf6f8" (UID: "ad113518-40b2-4d26-b886-a7f99afdf6f8"). InnerVolumeSpecName "kube-api-access-4z8ld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:13:07.731925 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.711130    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd9b0713-a9c7-4344-8b32-487d235f9a72-istio-token" pod "bd9b0713-a9c7-4344-8b32-487d235f9a72" (UID: "bd9b0713-a9c7-4344-8b32-487d235f9a72"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:13:07.731925 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.716158    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad113518-40b2-4d26-b886-a7f99afdf6f8-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:07.731925 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.716185    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd9b0713-a9c7-4344-8b32-487d235f9a72-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:07.731925 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.716203    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad113518-40b2-4d26-b886-a7f99afdf6f8-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:07.731925 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.716224    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8997a5-bfb9-4517-bb5e-632f8639e44d-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:07.731925 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.716242    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad113518-40b2-4d26-b886-a7f99afdf6f8-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:07.731925 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.716256    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd9b0713-a9c7-4344-8b32-487d235f9a72-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:07.731925 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.716271    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd9b0713-a9c7-4344-8b32-487d235f9a72-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:07.731925 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.716291    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad113518-40b2-4d26-b886-a7f99afdf6f8-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:07.731925 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.716308    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e8997a5-bfb9-4517-bb5e-632f8639e44d-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:07.731925 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.716322    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd9b0713-a9c7-4344-8b32-487d235f9a72-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:07.731925 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.716337    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z8ld\" (UniqueName: \"kubernetes.io/projected/ad113518-40b2-4d26-b886-a7f99afdf6f8-kube-api-access-4z8ld\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:07.731925 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.716358    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd9b0713-a9c7-4344-8b32-487d235f9a72-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:07.731925 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.716372    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e8997a5-bfb9-4517-bb5e-632f8639e44d-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:07.731925 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.716386    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad113518-40b2-4d26-b886-a7f99afdf6f8-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:07.731925 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.716401    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8997a5-bfb9-4517-bb5e-632f8639e44d-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:07.731925 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.716424    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whghb\" (UniqueName: \"kubernetes.io/projected/4e8997a5-bfb9-4517-bb5e-632f8639e44d-kube-api-access-whghb\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:07.731925 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.716439    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e8997a5-bfb9-4517-bb5e-632f8639e44d-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:07.731925 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.716456    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e8997a5-bfb9-4517-bb5e-632f8639e44d-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:07.731925 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.716471    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd9b0713-a9c7-4344-8b32-487d235f9a72-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:07.731925 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.716491    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad113518-40b2-4d26-b886-a7f99afdf6f8-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:07.731925 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.716507    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e8997a5-bfb9-4517-bb5e-632f8639e44d-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:07.731925 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.716522    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e8997a5-bfb9-4517-bb5e-632f8639e44d-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:07.731925 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.716540    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd9b0713-a9c7-4344-8b32-487d235f9a72-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:07.731925 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.716556    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd9b0713-a9c7-4344-8b32-487d235f9a72-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:07.731925 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.716572    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cnqk\" (UniqueName: \"kubernetes.io/projected/bd9b0713-a9c7-4344-8b32-487d235f9a72-kube-api-access-9cnqk\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:07.731925 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.716589    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad113518-40b2-4d26-b886-a7f99afdf6f8-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:07.733746 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.733428    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad113518-40b2-4d26-b886-a7f99afdf6f8-istio-podinfo" pod "ad113518-40b2-4d26-b886-a7f99afdf6f8" (UID: "ad113518-40b2-4d26-b886-a7f99afdf6f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:13:07.735204 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.735176    5315 generic.go:356] "Generic (PLEG): container finished" podID="ad113518-40b2-4d26-b886-a7f99afdf6f8" containerID="e6beb19892d462933e2480386ca6c9fb41d3fb186fa0a6e2879c86a73ba7a53d" exitCode=0
Nov 03 18:13:07.735542 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.735490    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff"
Nov 03 18:13:07.736189 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.736164    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff" event={"ID":"ad113518-40b2-4d26-b886-a7f99afdf6f8","Type":"ContainerDied","Data":"e6beb19892d462933e2480386ca6c9fb41d3fb186fa0a6e2879c86a73ba7a53d"}
Nov 03 18:13:07.736268 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.736237    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff" event={"ID":"ad113518-40b2-4d26-b886-a7f99afdf6f8","Type":"ContainerDied","Data":"f8f0a34dc15635ae895c546915e5fb03efbad3679dce44c7acc3a5b26640b737"}
Nov 03 18:13:07.736327 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.736308    5315 scope.go:122] "RemoveContainer" containerID="e6beb19892d462933e2480386ca6c9fb41d3fb186fa0a6e2879c86a73ba7a53d"
Nov 03 18:13:07.739321 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.739287    5315 generic.go:356] "Generic (PLEG): container finished" podID="bd9b0713-a9c7-4344-8b32-487d235f9a72" containerID="043eaf103f43bb3b1032c7ba51943da1b159dbd12fd55c4ce69d1f58a2998d23" exitCode=0
Nov 03 18:13:07.739515 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.739369    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj" event={"ID":"bd9b0713-a9c7-4344-8b32-487d235f9a72","Type":"ContainerDied","Data":"043eaf103f43bb3b1032c7ba51943da1b159dbd12fd55c4ce69d1f58a2998d23"}
Nov 03 18:13:07.744995 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.739379    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj"
Nov 03 18:13:07.747339 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.747235    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj" event={"ID":"bd9b0713-a9c7-4344-8b32-487d235f9a72","Type":"ContainerDied","Data":"419b63653f48325ff899a3d1f5a17153275403b2696ce6ebc1e57cec7f63f109"}
Nov 03 18:13:07.748646 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.748597    5315 generic.go:356] "Generic (PLEG): container finished" podID="59f3868f-6b95-4d02-afe7-4e5228cf37c3" containerID="e9f733f4fd8b9f3f646c0164b8eb5616c00eca87ff3c03c464c9217d024cc71f" exitCode=0
Nov 03 18:13:07.748845 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.748827    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp"
Nov 03 18:13:07.749707 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.749096    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp" event={"ID":"59f3868f-6b95-4d02-afe7-4e5228cf37c3","Type":"ContainerDied","Data":"e9f733f4fd8b9f3f646c0164b8eb5616c00eca87ff3c03c464c9217d024cc71f"}
Nov 03 18:13:07.749707 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.749156    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp" event={"ID":"59f3868f-6b95-4d02-afe7-4e5228cf37c3","Type":"ContainerDied","Data":"c7424cb8369383e8bc1200d6fe00b6f49a5c3bd44a0acead0723bd2511374fa0"}
Nov 03 18:13:07.766622 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.758883    5315 generic.go:356] "Generic (PLEG): container finished" podID="4e8997a5-bfb9-4517-bb5e-632f8639e44d" containerID="2975927156f1c80407a69ee06540379ec4bdcc7d0e458453e8d773bdaaa70515" exitCode=0
Nov 03 18:13:07.766622 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.758963    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45"
Nov 03 18:13:07.766622 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.758968    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45" event={"ID":"4e8997a5-bfb9-4517-bb5e-632f8639e44d","Type":"ContainerDied","Data":"2975927156f1c80407a69ee06540379ec4bdcc7d0e458453e8d773bdaaa70515"}
Nov 03 18:13:07.766622 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.759008    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45" event={"ID":"4e8997a5-bfb9-4517-bb5e-632f8639e44d","Type":"ContainerDied","Data":"c06b0303795abc7bbb3d8c25a5cff37c3b88c67e5a48389732485dd3a4b68046"}
Nov 03 18:13:07.776044 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.769906    5315 scope.go:122] "RemoveContainer" containerID="e6beb19892d462933e2480386ca6c9fb41d3fb186fa0a6e2879c86a73ba7a53d"
Nov 03 18:13:07.777786 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:13:07.777600    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6beb19892d462933e2480386ca6c9fb41d3fb186fa0a6e2879c86a73ba7a53d\": container with ID starting with e6beb19892d462933e2480386ca6c9fb41d3fb186fa0a6e2879c86a73ba7a53d not found: ID does not exist" containerID="e6beb19892d462933e2480386ca6c9fb41d3fb186fa0a6e2879c86a73ba7a53d"
Nov 03 18:13:07.777786 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.777686    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6beb19892d462933e2480386ca6c9fb41d3fb186fa0a6e2879c86a73ba7a53d"} err="failed to get container status \"e6beb19892d462933e2480386ca6c9fb41d3fb186fa0a6e2879c86a73ba7a53d\": rpc error: code = NotFound desc = could not find container \"e6beb19892d462933e2480386ca6c9fb41d3fb186fa0a6e2879c86a73ba7a53d\": container with ID starting with e6beb19892d462933e2480386ca6c9fb41d3fb186fa0a6e2879c86a73ba7a53d not found: ID does not exist"
Nov 03 18:13:07.777786 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.777711    5315 scope.go:122] "RemoveContainer" containerID="043eaf103f43bb3b1032c7ba51943da1b159dbd12fd55c4ce69d1f58a2998d23"
Nov 03 18:13:07.790796 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.790778    5315 scope.go:122] "RemoveContainer" containerID="043eaf103f43bb3b1032c7ba51943da1b159dbd12fd55c4ce69d1f58a2998d23"
Nov 03 18:13:07.791084 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:13:07.791059    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"043eaf103f43bb3b1032c7ba51943da1b159dbd12fd55c4ce69d1f58a2998d23\": container with ID starting with 043eaf103f43bb3b1032c7ba51943da1b159dbd12fd55c4ce69d1f58a2998d23 not found: ID does not exist" containerID="043eaf103f43bb3b1032c7ba51943da1b159dbd12fd55c4ce69d1f58a2998d23"
Nov 03 18:13:07.791161 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.791095    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"043eaf103f43bb3b1032c7ba51943da1b159dbd12fd55c4ce69d1f58a2998d23"} err="failed to get container status \"043eaf103f43bb3b1032c7ba51943da1b159dbd12fd55c4ce69d1f58a2998d23\": rpc error: code = NotFound desc = could not find container \"043eaf103f43bb3b1032c7ba51943da1b159dbd12fd55c4ce69d1f58a2998d23\": container with ID starting with 043eaf103f43bb3b1032c7ba51943da1b159dbd12fd55c4ce69d1f58a2998d23 not found: ID does not exist"
Nov 03 18:13:07.791161 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.791119    5315 scope.go:122] "RemoveContainer" containerID="e9f733f4fd8b9f3f646c0164b8eb5616c00eca87ff3c03c464c9217d024cc71f"
Nov 03 18:13:07.801164 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.801143    5315 scope.go:122] "RemoveContainer" containerID="e9f733f4fd8b9f3f646c0164b8eb5616c00eca87ff3c03c464c9217d024cc71f"
Nov 03 18:13:07.801464 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:13:07.801443    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9f733f4fd8b9f3f646c0164b8eb5616c00eca87ff3c03c464c9217d024cc71f\": container with ID starting with e9f733f4fd8b9f3f646c0164b8eb5616c00eca87ff3c03c464c9217d024cc71f not found: ID does not exist" containerID="e9f733f4fd8b9f3f646c0164b8eb5616c00eca87ff3c03c464c9217d024cc71f"
Nov 03 18:13:07.801527 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.801473    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9f733f4fd8b9f3f646c0164b8eb5616c00eca87ff3c03c464c9217d024cc71f"} err="failed to get container status \"e9f733f4fd8b9f3f646c0164b8eb5616c00eca87ff3c03c464c9217d024cc71f\": rpc error: code = NotFound desc = could not find container \"e9f733f4fd8b9f3f646c0164b8eb5616c00eca87ff3c03c464c9217d024cc71f\": container with ID starting with e9f733f4fd8b9f3f646c0164b8eb5616c00eca87ff3c03c464c9217d024cc71f not found: ID does not exist"
Nov 03 18:13:07.801527 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.801489    5315 scope.go:122] "RemoveContainer" containerID="2975927156f1c80407a69ee06540379ec4bdcc7d0e458453e8d773bdaaa70515"
Nov 03 18:13:07.811703 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.811677    5315 scope.go:122] "RemoveContainer" containerID="6afcb66d3e49bf9e99b751a2306aa56d55b92c13a671ac2d80588a967d564ce2"
Nov 03 18:13:07.818543 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.818520    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad113518-40b2-4d26-b886-a7f99afdf6f8-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:07.821084 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.821067    5315 scope.go:122] "RemoveContainer" containerID="2975927156f1c80407a69ee06540379ec4bdcc7d0e458453e8d773bdaaa70515"
Nov 03 18:13:07.821377 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:13:07.821355    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2975927156f1c80407a69ee06540379ec4bdcc7d0e458453e8d773bdaaa70515\": container with ID starting with 2975927156f1c80407a69ee06540379ec4bdcc7d0e458453e8d773bdaaa70515 not found: ID does not exist" containerID="2975927156f1c80407a69ee06540379ec4bdcc7d0e458453e8d773bdaaa70515"
Nov 03 18:13:07.821437 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.821385    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2975927156f1c80407a69ee06540379ec4bdcc7d0e458453e8d773bdaaa70515"} err="failed to get container status \"2975927156f1c80407a69ee06540379ec4bdcc7d0e458453e8d773bdaaa70515\": rpc error: code = NotFound desc = could not find container \"2975927156f1c80407a69ee06540379ec4bdcc7d0e458453e8d773bdaaa70515\": container with ID starting with 2975927156f1c80407a69ee06540379ec4bdcc7d0e458453e8d773bdaaa70515 not found: ID does not exist"
Nov 03 18:13:07.821437 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.821410    5315 scope.go:122] "RemoveContainer" containerID="6afcb66d3e49bf9e99b751a2306aa56d55b92c13a671ac2d80588a967d564ce2"
Nov 03 18:13:07.821835 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:13:07.821811    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6afcb66d3e49bf9e99b751a2306aa56d55b92c13a671ac2d80588a967d564ce2\": container with ID starting with 6afcb66d3e49bf9e99b751a2306aa56d55b92c13a671ac2d80588a967d564ce2 not found: ID does not exist" containerID="6afcb66d3e49bf9e99b751a2306aa56d55b92c13a671ac2d80588a967d564ce2"
Nov 03 18:13:07.821968 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:07.821845    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6afcb66d3e49bf9e99b751a2306aa56d55b92c13a671ac2d80588a967d564ce2"} err="failed to get container status \"6afcb66d3e49bf9e99b751a2306aa56d55b92c13a671ac2d80588a967d564ce2\": rpc error: code = NotFound desc = could not find container \"6afcb66d3e49bf9e99b751a2306aa56d55b92c13a671ac2d80588a967d564ce2\": container with ID starting with 6afcb66d3e49bf9e99b751a2306aa56d55b92c13a671ac2d80588a967d564ce2 not found: ID does not exist"
Nov 03 18:13:08.140592 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:08.140556    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45"]
Nov 03 18:13:08.154121 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:08.154082    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-czp45"]
Nov 03 18:13:08.248816 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:08.248733    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp"]
Nov 03 18:13:08.254800 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:08.254768    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-2bvvp"]
Nov 03 18:13:08.356827 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:08.356794    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff"]
Nov 03 18:13:08.363554 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:08.363521    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-lvkff"]
Nov 03 18:13:08.467023 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:08.466983    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj"]
Nov 03 18:13:08.471704 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:08.471675    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-557mj"]
Nov 03 18:13:09.105591 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:09.105543    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e8997a5-bfb9-4517-bb5e-632f8639e44d" path="/var/lib/kubelet/pods/4e8997a5-bfb9-4517-bb5e-632f8639e44d/volumes"
Nov 03 18:13:09.106202 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:09.106179    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59f3868f-6b95-4d02-afe7-4e5228cf37c3" path="/var/lib/kubelet/pods/59f3868f-6b95-4d02-afe7-4e5228cf37c3/volumes"
Nov 03 18:13:09.106719 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:09.106700    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad113518-40b2-4d26-b886-a7f99afdf6f8" path="/var/lib/kubelet/pods/ad113518-40b2-4d26-b886-a7f99afdf6f8/volumes"
Nov 03 18:13:09.107697 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:09.107496    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd9b0713-a9c7-4344-8b32-487d235f9a72" path="/var/lib/kubelet/pods/bd9b0713-a9c7-4344-8b32-487d235f9a72/volumes"
Nov 03 18:13:13.472092 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.472009    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7347-a6fd67"]
Nov 03 18:13:13.472776 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.472755    5315 cpu_manager.go:414] "containerMap: removing container" podUID="1e4ca152-3632-485f-94b2-4f430cbdfe4c" containerName="istio-proxy"
Nov 03 18:13:13.472776 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.472777    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e4ca152-3632-485f-94b2-4f430cbdfe4c" containerName="istio-proxy"
Nov 03 18:13:13.472914 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.472796    5315 cpu_manager.go:414] "containerMap: removing container" podUID="79050b41-0f3b-4fc4-a8e8-50316359ac33" containerName="istio-proxy"
Nov 03 18:13:13.472914 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.472801    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79050b41-0f3b-4fc4-a8e8-50316359ac33" containerName="istio-proxy"
Nov 03 18:13:13.472914 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.472814    5315 cpu_manager.go:414] "containerMap: removing container" podUID="bd9b0713-a9c7-4344-8b32-487d235f9a72" containerName="istio-proxy"
Nov 03 18:13:13.472914 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.472822    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd9b0713-a9c7-4344-8b32-487d235f9a72" containerName="istio-proxy"
Nov 03 18:13:13.472914 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.472839    5315 cpu_manager.go:414] "containerMap: removing container" podUID="858e4f16-d2d2-4ba6-9e75-b81fe8ad5674" containerName="istio-proxy"
Nov 03 18:13:13.472914 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.472846    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="858e4f16-d2d2-4ba6-9e75-b81fe8ad5674" containerName="istio-proxy"
Nov 03 18:13:13.472914 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.472859    5315 cpu_manager.go:414] "containerMap: removing container" podUID="25a8498d-e23e-44a9-81fc-3170b124ad13" containerName="istio-proxy"
Nov 03 18:13:13.472914 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.472864    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25a8498d-e23e-44a9-81fc-3170b124ad13" containerName="istio-proxy"
Nov 03 18:13:13.472914 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.472871    5315 cpu_manager.go:414] "containerMap: removing container" podUID="2b453729-cac0-4839-bdfa-0da04e59c1bc" containerName="istio-proxy"
Nov 03 18:13:13.472914 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.472876    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b453729-cac0-4839-bdfa-0da04e59c1bc" containerName="istio-proxy"
Nov 03 18:13:13.472914 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.472884    5315 cpu_manager.go:414] "containerMap: removing container" podUID="4e8997a5-bfb9-4517-bb5e-632f8639e44d" containerName="istio-proxy"
Nov 03 18:13:13.472914 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.472890    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e8997a5-bfb9-4517-bb5e-632f8639e44d" containerName="istio-proxy"
Nov 03 18:13:13.472914 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.472898    5315 cpu_manager.go:414] "containerMap: removing container" podUID="04fd7ef5-cf3c-4ed7-963d-e03294eaf327" containerName="istio-proxy"
Nov 03 18:13:13.472914 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.472905    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04fd7ef5-cf3c-4ed7-963d-e03294eaf327" containerName="istio-proxy"
Nov 03 18:13:13.472914 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.472918    5315 cpu_manager.go:414] "containerMap: removing container" podUID="ad113518-40b2-4d26-b886-a7f99afdf6f8" containerName="istio-proxy"
Nov 03 18:13:13.473340 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.472923    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad113518-40b2-4d26-b886-a7f99afdf6f8" containerName="istio-proxy"
Nov 03 18:13:13.473340 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.472936    5315 cpu_manager.go:414] "containerMap: removing container" podUID="4e8997a5-bfb9-4517-bb5e-632f8639e44d" containerName="istio-proxy"
Nov 03 18:13:13.473340 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.472940    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e8997a5-bfb9-4517-bb5e-632f8639e44d" containerName="istio-proxy"
Nov 03 18:13:13.473340 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.472947    5315 cpu_manager.go:414] "containerMap: removing container" podUID="25a8498d-e23e-44a9-81fc-3170b124ad13" containerName="istio-proxy"
Nov 03 18:13:13.473340 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.472951    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25a8498d-e23e-44a9-81fc-3170b124ad13" containerName="istio-proxy"
Nov 03 18:13:13.473340 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.472961    5315 cpu_manager.go:414] "containerMap: removing container" podUID="375f4e00-8c81-4de5-97bf-9be54ea028ed" containerName="istio-proxy"
Nov 03 18:13:13.473340 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.472965    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="375f4e00-8c81-4de5-97bf-9be54ea028ed" containerName="istio-proxy"
Nov 03 18:13:13.473340 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.472973    5315 cpu_manager.go:414] "containerMap: removing container" podUID="57d6511c-f9f2-4ca7-9bfb-6475dd17f94b" containerName="istio-proxy"
Nov 03 18:13:13.473340 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.472977    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57d6511c-f9f2-4ca7-9bfb-6475dd17f94b" containerName="istio-proxy"
Nov 03 18:13:13.473340 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.472987    5315 cpu_manager.go:414] "containerMap: removing container" podUID="59f3868f-6b95-4d02-afe7-4e5228cf37c3" containerName="istio-proxy"
Nov 03 18:13:13.473340 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.472991    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59f3868f-6b95-4d02-afe7-4e5228cf37c3" containerName="istio-proxy"
Nov 03 18:13:13.473340 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.473087    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="375f4e00-8c81-4de5-97bf-9be54ea028ed" containerName="istio-proxy"
Nov 03 18:13:13.473340 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.473098    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="57d6511c-f9f2-4ca7-9bfb-6475dd17f94b" containerName="istio-proxy"
Nov 03 18:13:13.473340 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.473104    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="59f3868f-6b95-4d02-afe7-4e5228cf37c3" containerName="istio-proxy"
Nov 03 18:13:13.473340 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.473112    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e4ca152-3632-485f-94b2-4f430cbdfe4c" containerName="istio-proxy"
Nov 03 18:13:13.473340 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.473118    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="38acc846-2251-4000-ae98-e5a2492da6a6" containerName="istio-proxy"
Nov 03 18:13:13.473340 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.473124    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="79050b41-0f3b-4fc4-a8e8-50316359ac33" containerName="istio-proxy"
Nov 03 18:13:13.473340 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.473131    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd9b0713-a9c7-4344-8b32-487d235f9a72" containerName="istio-proxy"
Nov 03 18:13:13.473340 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.473137    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="858e4f16-d2d2-4ba6-9e75-b81fe8ad5674" containerName="istio-proxy"
Nov 03 18:13:13.473340 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.473145    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="25a8498d-e23e-44a9-81fc-3170b124ad13" containerName="istio-proxy"
Nov 03 18:13:13.473340 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.473150    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b453729-cac0-4839-bdfa-0da04e59c1bc" containerName="istio-proxy"
Nov 03 18:13:13.473340 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.473156    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e8997a5-bfb9-4517-bb5e-632f8639e44d" containerName="istio-proxy"
Nov 03 18:13:13.473340 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.473160    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="04fd7ef5-cf3c-4ed7-963d-e03294eaf327" containerName="istio-proxy"
Nov 03 18:13:13.473340 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.473166    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad113518-40b2-4d26-b886-a7f99afdf6f8" containerName="istio-proxy"
Nov 03 18:13:13.473340 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.473173    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e8997a5-bfb9-4517-bb5e-632f8639e44d" containerName="istio-proxy"
Nov 03 18:13:13.473340 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.473180    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="25a8498d-e23e-44a9-81fc-3170b124ad13" containerName="istio-proxy"
Nov 03 18:13:13.476990 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.476967    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7347-a6fd67"
Nov 03 18:13:13.525413 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.525382    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-447v9\""
Nov 03 18:13:13.577021 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.576977    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7347-a6fd67"]
Nov 03 18:13:13.578158 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.578125    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg2xx\" (UniqueName: \"kubernetes.io/projected/6516e404-f788-49ce-82ac-d819701ce0cf-kube-api-access-cg2xx\") pod \"test-curl-7347-a6fd67\" (UID: \"6516e404-f788-49ce-82ac-d819701ce0cf\") " pod="opendatahub/test-curl-7347-a6fd67"
Nov 03 18:13:13.679261 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.679226    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg2xx\" (UniqueName: \"kubernetes.io/projected/6516e404-f788-49ce-82ac-d819701ce0cf-kube-api-access-cg2xx\") pod \"test-curl-7347-a6fd67\" (UID: \"6516e404-f788-49ce-82ac-d819701ce0cf\") " pod="opendatahub/test-curl-7347-a6fd67"
Nov 03 18:13:13.735973 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.735907    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg2xx\" (UniqueName: \"kubernetes.io/projected/6516e404-f788-49ce-82ac-d819701ce0cf-kube-api-access-cg2xx\") pod \"test-curl-7347-a6fd67\" (UID: \"6516e404-f788-49ce-82ac-d819701ce0cf\") " pod="opendatahub/test-curl-7347-a6fd67"
Nov 03 18:13:13.800475 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:13.800434    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7347-a6fd67"
Nov 03 18:13:14.175271 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:14.175235    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7347-a6fd67"]
Nov 03 18:13:14.180924 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:13:14.180891    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6516e404_f788_49ce_82ac_d819701ce0cf.slice/crio-b502e0a2c49386a1cdf6f35c0af3d6e2b5987e9abdf2b6587b1b8a7a4687167a WatchSource:0}: Error finding container b502e0a2c49386a1cdf6f35c0af3d6e2b5987e9abdf2b6587b1b8a7a4687167a: Status 404 returned error can't find the container with id b502e0a2c49386a1cdf6f35c0af3d6e2b5987e9abdf2b6587b1b8a7a4687167a
Nov 03 18:13:14.843412 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:14.843370    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7347-a6fd67" event={"ID":"6516e404-f788-49ce-82ac-d819701ce0cf","Type":"ContainerStarted","Data":"9edaa18fd4fcd53f4e3d446a50e7b3b6fd28f1ef2ea172fb6748543a33045907"}
Nov 03 18:13:14.843412 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:14.843417    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7347-a6fd67" event={"ID":"6516e404-f788-49ce-82ac-d819701ce0cf","Type":"ContainerStarted","Data":"b502e0a2c49386a1cdf6f35c0af3d6e2b5987e9abdf2b6587b1b8a7a4687167a"}
Nov 03 18:13:14.950552 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:14.950496    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7347-a6fd67" podStartSLOduration=1.950480575 podStartE2EDuration="1.950480575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:13:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:13:14.948001671 +0000 UTC m=+1633.964199733" watchObservedRunningTime="2026-11-03 18:13:14.950480575 +0000 UTC m=+1633.966678643"
Nov 03 18:13:15.046492 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:15.046450    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f26ceb6a-4rd6s"]
Nov 03 18:13:15.050383 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:15.050352    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f26ceb6a-4rd6s"]
Nov 03 18:13:15.109055 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:15.108973    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee361c55-2d7d-4b37-b1c5-3d4f29868fd4" path="/var/lib/kubelet/pods/ee361c55-2d7d-4b37-b1c5-3d4f29868fd4/volumes"
Nov 03 18:13:16.409686 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:16.409647    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7347-a6fd67"]
Nov 03 18:13:16.414140 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:16.414109    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7347-a6fd67"]
Nov 03 18:13:16.574123 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:16.574085    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7347-e1b8ef"]
Nov 03 18:13:16.574948 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:16.574918    5315 cpu_manager.go:414] "containerMap: removing container" podUID="6516e404-f788-49ce-82ac-d819701ce0cf" containerName="test-curl-7347-a6fd67"
Nov 03 18:13:16.574948 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:16.574942    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6516e404-f788-49ce-82ac-d819701ce0cf" containerName="test-curl-7347-a6fd67"
Nov 03 18:13:16.575090 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:16.575080    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="6516e404-f788-49ce-82ac-d819701ce0cf" containerName="test-curl-7347-a6fd67"
Nov 03 18:13:16.577769 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:16.577748    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7347-e1b8ef"
Nov 03 18:13:16.636087 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:16.636051    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7347-e1b8ef"]
Nov 03 18:13:16.723645 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:16.723539    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brfvz\" (UniqueName: \"kubernetes.io/projected/bd8c47da-62ba-4d3d-b257-8acedccdce77-kube-api-access-brfvz\") pod \"test-curl-7347-e1b8ef\" (UID: \"bd8c47da-62ba-4d3d-b257-8acedccdce77\") " pod="opendatahub/test-curl-7347-e1b8ef"
Nov 03 18:13:16.824918 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:16.824880    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brfvz\" (UniqueName: \"kubernetes.io/projected/bd8c47da-62ba-4d3d-b257-8acedccdce77-kube-api-access-brfvz\") pod \"test-curl-7347-e1b8ef\" (UID: \"bd8c47da-62ba-4d3d-b257-8acedccdce77\") " pod="opendatahub/test-curl-7347-e1b8ef"
Nov 03 18:13:16.862779 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:16.862734    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7347-a6fd67" podUID="6516e404-f788-49ce-82ac-d819701ce0cf" containerName="test-curl-7347-a6fd67" containerID="cri-o://9edaa18fd4fcd53f4e3d446a50e7b3b6fd28f1ef2ea172fb6748543a33045907" gracePeriod=2
Nov 03 18:13:16.881884 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:16.881848    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brfvz\" (UniqueName: \"kubernetes.io/projected/bd8c47da-62ba-4d3d-b257-8acedccdce77-kube-api-access-brfvz\") pod \"test-curl-7347-e1b8ef\" (UID: \"bd8c47da-62ba-4d3d-b257-8acedccdce77\") " pod="opendatahub/test-curl-7347-e1b8ef"
Nov 03 18:13:16.889235 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:16.889209    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7347-e1b8ef"
Nov 03 18:13:16.910260 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:13:16.910215    5315 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7347-a6fd67\" is forbidden: User \"system:node:ip-10-0-128-204.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-204.us-west-2.compute.internal' and this object" podUID="6516e404-f788-49ce-82ac-d819701ce0cf" pod="opendatahub/test-curl-7347-a6fd67"
Nov 03 18:13:17.282755 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:17.282669    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7347-e1b8ef"]
Nov 03 18:13:17.288940 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:13:17.288909    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd8c47da_62ba_4d3d_b257_8acedccdce77.slice/crio-dee4632e5d38970ecb7aa1aef3fe239f6c2a0c92c8ca3bd035448a15da594058 WatchSource:0}: Error finding container dee4632e5d38970ecb7aa1aef3fe239f6c2a0c92c8ca3bd035448a15da594058: Status 404 returned error can't find the container with id dee4632e5d38970ecb7aa1aef3fe239f6c2a0c92c8ca3bd035448a15da594058
Nov 03 18:13:17.871840 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:17.871800    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7347-e1b8ef" event={"ID":"bd8c47da-62ba-4d3d-b257-8acedccdce77","Type":"ContainerStarted","Data":"5436a2b19e9c222279555aed1559bc0afea5a2d7c871918723c7d7f44cb71bd4"}
Nov 03 18:13:17.871840 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:17.871848    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7347-e1b8ef" event={"ID":"bd8c47da-62ba-4d3d-b257-8acedccdce77","Type":"ContainerStarted","Data":"dee4632e5d38970ecb7aa1aef3fe239f6c2a0c92c8ca3bd035448a15da594058"}
Nov 03 18:13:17.977570 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:17.977513    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7347-e1b8ef" podStartSLOduration=1.977498507 podStartE2EDuration="1.977498507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:13:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:13:17.976340401 +0000 UTC m=+1636.992538470" watchObservedRunningTime="2026-11-03 18:13:17.977498507 +0000 UTC m=+1636.993696576"
Nov 03 18:13:19.152343 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:19.152317    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7347-a6fd67"
Nov 03 18:13:19.273605 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:19.273574    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6516e404-f788-49ce-82ac-d819701ce0cf-kube-api-access-cg2xx\" (UniqueName: \"kubernetes.io/projected/6516e404-f788-49ce-82ac-d819701ce0cf-kube-api-access-cg2xx\") pod \"6516e404-f788-49ce-82ac-d819701ce0cf\" (UID: \"6516e404-f788-49ce-82ac-d819701ce0cf\") "
Nov 03 18:13:19.275694 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:19.275669    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6516e404-f788-49ce-82ac-d819701ce0cf-kube-api-access-cg2xx" pod "6516e404-f788-49ce-82ac-d819701ce0cf" (UID: "6516e404-f788-49ce-82ac-d819701ce0cf"). InnerVolumeSpecName "kube-api-access-cg2xx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:13:19.376512 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:19.376428    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg2xx\" (UniqueName: \"kubernetes.io/projected/6516e404-f788-49ce-82ac-d819701ce0cf-kube-api-access-cg2xx\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:19.587649 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:19.586202    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7347-e1b8ef"]
Nov 03 18:13:19.594579 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:19.594542    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7347-e1b8ef"]
Nov 03 18:13:19.764724 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:19.764599    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7347-efc448"]
Nov 03 18:13:19.765499 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:19.765473    5315 cpu_manager.go:414] "containerMap: removing container" podUID="bd8c47da-62ba-4d3d-b257-8acedccdce77" containerName="test-curl-7347-e1b8ef"
Nov 03 18:13:19.765499 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:19.765498    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd8c47da-62ba-4d3d-b257-8acedccdce77" containerName="test-curl-7347-e1b8ef"
Nov 03 18:13:19.765732 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:19.765716    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd8c47da-62ba-4d3d-b257-8acedccdce77" containerName="test-curl-7347-e1b8ef"
Nov 03 18:13:19.768178 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:19.768156    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7347-efc448"
Nov 03 18:13:19.826764 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:19.826726    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7347-efc448"]
Nov 03 18:13:19.884295 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:19.884254    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djdkb\" (UniqueName: \"kubernetes.io/projected/dce1aa3b-8168-4aad-b1d8-2f4a7ffa9636-kube-api-access-djdkb\") pod \"test-curl-7347-efc448\" (UID: \"dce1aa3b-8168-4aad-b1d8-2f4a7ffa9636\") " pod="opendatahub/test-curl-7347-efc448"
Nov 03 18:13:19.890791 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:19.890763    5315 generic.go:356] "Generic (PLEG): container finished" podID="6516e404-f788-49ce-82ac-d819701ce0cf" containerID="9edaa18fd4fcd53f4e3d446a50e7b3b6fd28f1ef2ea172fb6748543a33045907" exitCode=137
Nov 03 18:13:19.890946 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:19.890814    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7347-a6fd67"
Nov 03 18:13:19.890946 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:19.890852    5315 scope.go:122] "RemoveContainer" containerID="9edaa18fd4fcd53f4e3d446a50e7b3b6fd28f1ef2ea172fb6748543a33045907"
Nov 03 18:13:19.891072 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:19.891008    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7347-e1b8ef" podUID="bd8c47da-62ba-4d3d-b257-8acedccdce77" containerName="test-curl-7347-e1b8ef" containerID="cri-o://5436a2b19e9c222279555aed1559bc0afea5a2d7c871918723c7d7f44cb71bd4" gracePeriod=2
Nov 03 18:13:19.901177 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:19.901155    5315 scope.go:122] "RemoveContainer" containerID="9edaa18fd4fcd53f4e3d446a50e7b3b6fd28f1ef2ea172fb6748543a33045907"
Nov 03 18:13:19.901475 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:13:19.901454    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9edaa18fd4fcd53f4e3d446a50e7b3b6fd28f1ef2ea172fb6748543a33045907\": container with ID starting with 9edaa18fd4fcd53f4e3d446a50e7b3b6fd28f1ef2ea172fb6748543a33045907 not found: ID does not exist" containerID="9edaa18fd4fcd53f4e3d446a50e7b3b6fd28f1ef2ea172fb6748543a33045907"
Nov 03 18:13:19.901522 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:19.901484    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9edaa18fd4fcd53f4e3d446a50e7b3b6fd28f1ef2ea172fb6748543a33045907"} err="failed to get container status \"9edaa18fd4fcd53f4e3d446a50e7b3b6fd28f1ef2ea172fb6748543a33045907\": rpc error: code = NotFound desc = could not find container \"9edaa18fd4fcd53f4e3d446a50e7b3b6fd28f1ef2ea172fb6748543a33045907\": container with ID starting with 9edaa18fd4fcd53f4e3d446a50e7b3b6fd28f1ef2ea172fb6748543a33045907 not found: ID does not exist"
Nov 03 18:13:19.938328 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:13:19.938291    5315 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7347-e1b8ef\" is forbidden: User \"system:node:ip-10-0-128-204.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-204.us-west-2.compute.internal' and this object" podUID="bd8c47da-62ba-4d3d-b257-8acedccdce77" pod="opendatahub/test-curl-7347-e1b8ef"
Nov 03 18:13:19.985601 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:13:19.985561    5315 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7347-a6fd67\" is forbidden: User \"system:node:ip-10-0-128-204.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-204.us-west-2.compute.internal' and this object" podUID="6516e404-f788-49ce-82ac-d819701ce0cf" pod="opendatahub/test-curl-7347-a6fd67"
Nov 03 18:13:19.985768 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:19.985725    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djdkb\" (UniqueName: \"kubernetes.io/projected/dce1aa3b-8168-4aad-b1d8-2f4a7ffa9636-kube-api-access-djdkb\") pod \"test-curl-7347-efc448\" (UID: \"dce1aa3b-8168-4aad-b1d8-2f4a7ffa9636\") " pod="opendatahub/test-curl-7347-efc448"
Nov 03 18:13:20.032911 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:13:20.032867    5315 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7347-e1b8ef\" is forbidden: User \"system:node:ip-10-0-128-204.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-204.us-west-2.compute.internal' and this object" podUID="bd8c47da-62ba-4d3d-b257-8acedccdce77" pod="opendatahub/test-curl-7347-e1b8ef"
Nov 03 18:13:20.042565 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:20.042536    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djdkb\" (UniqueName: \"kubernetes.io/projected/dce1aa3b-8168-4aad-b1d8-2f4a7ffa9636-kube-api-access-djdkb\") pod \"test-curl-7347-efc448\" (UID: \"dce1aa3b-8168-4aad-b1d8-2f4a7ffa9636\") " pod="opendatahub/test-curl-7347-efc448"
Nov 03 18:13:20.079800 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:20.079766    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7347-efc448"
Nov 03 18:13:20.452003 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:20.451912    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7347-efc448"]
Nov 03 18:13:20.457502 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:13:20.457469    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddce1aa3b_8168_4aad_b1d8_2f4a7ffa9636.slice/crio-5554bb9cd55f32744163f06b84ae0603884faec7c788d6eb6b57c4e03df075d3 WatchSource:0}: Error finding container 5554bb9cd55f32744163f06b84ae0603884faec7c788d6eb6b57c4e03df075d3: Status 404 returned error can't find the container with id 5554bb9cd55f32744163f06b84ae0603884faec7c788d6eb6b57c4e03df075d3
Nov 03 18:13:20.537492 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:20.537462    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b6cb8403-v7kr7"]
Nov 03 18:13:20.541312 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:20.541289    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b6cb8403-v7kr7"
Nov 03 18:13:20.589664 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:20.589629    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:13:20.589820 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:20.589795    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:13:20.589863 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:20.589820    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:13:20.635654 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:20.635600    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:13:20.641590 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:20.641551    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b6cb8403-v7kr7"]
Nov 03 18:13:20.694648 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:13:20.694586    5315 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7347-a6fd67\" is forbidden: User \"system:node:ip-10-0-128-204.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-204.us-west-2.compute.internal' and this object" podUID="6516e404-f788-49ce-82ac-d819701ce0cf" pod="opendatahub/test-curl-7347-a6fd67"
Nov 03 18:13:20.695289 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:20.695268    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45a6b9f7-0d90-448f-a209-5f5517530e72-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-b6cb8403-v7kr7\" (UID: \"45a6b9f7-0d90-448f-a209-5f5517530e72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b6cb8403-v7kr7"
Nov 03 18:13:20.695463 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:20.695448    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r5fr\" (UniqueName: \"kubernetes.io/projected/45a6b9f7-0d90-448f-a209-5f5517530e72-kube-api-access-8r5fr\") pod \"maas-api-revoke-keys-e2e-mt-b6cb8403-v7kr7\" (UID: \"45a6b9f7-0d90-448f-a209-5f5517530e72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b6cb8403-v7kr7"
Nov 03 18:13:20.695544 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:20.695531    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45a6b9f7-0d90-448f-a209-5f5517530e72-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-b6cb8403-v7kr7\" (UID: \"45a6b9f7-0d90-448f-a209-5f5517530e72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b6cb8403-v7kr7"
Nov 03 18:13:20.742016 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:13:20.741917    5315 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7347-e1b8ef\" is forbidden: User \"system:node:ip-10-0-128-204.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-204.us-west-2.compute.internal' and this object" podUID="bd8c47da-62ba-4d3d-b257-8acedccdce77" pod="opendatahub/test-curl-7347-e1b8ef"
Nov 03 18:13:20.796411 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:20.796368    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45a6b9f7-0d90-448f-a209-5f5517530e72-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-b6cb8403-v7kr7\" (UID: \"45a6b9f7-0d90-448f-a209-5f5517530e72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b6cb8403-v7kr7"
Nov 03 18:13:20.796586 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:20.796440    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45a6b9f7-0d90-448f-a209-5f5517530e72-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-b6cb8403-v7kr7\" (UID: \"45a6b9f7-0d90-448f-a209-5f5517530e72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b6cb8403-v7kr7"
Nov 03 18:13:20.796642 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:20.796586    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r5fr\" (UniqueName: \"kubernetes.io/projected/45a6b9f7-0d90-448f-a209-5f5517530e72-kube-api-access-8r5fr\") pod \"maas-api-revoke-keys-e2e-mt-b6cb8403-v7kr7\" (UID: \"45a6b9f7-0d90-448f-a209-5f5517530e72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b6cb8403-v7kr7"
Nov 03 18:13:20.797190 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:20.797165    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45a6b9f7-0d90-448f-a209-5f5517530e72-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-b6cb8403-v7kr7\" (UID: \"45a6b9f7-0d90-448f-a209-5f5517530e72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b6cb8403-v7kr7"
Nov 03 18:13:20.798830 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:20.798813    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45a6b9f7-0d90-448f-a209-5f5517530e72-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-b6cb8403-v7kr7\" (UID: \"45a6b9f7-0d90-448f-a209-5f5517530e72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b6cb8403-v7kr7"
Nov 03 18:13:20.853956 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:20.853909    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r5fr\" (UniqueName: \"kubernetes.io/projected/45a6b9f7-0d90-448f-a209-5f5517530e72-kube-api-access-8r5fr\") pod \"maas-api-revoke-keys-e2e-mt-b6cb8403-v7kr7\" (UID: \"45a6b9f7-0d90-448f-a209-5f5517530e72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b6cb8403-v7kr7"
Nov 03 18:13:20.854197 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:20.854164    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b6cb8403-v7kr7"
Nov 03 18:13:20.903709 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:20.903668    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7347-efc448" event={"ID":"dce1aa3b-8168-4aad-b1d8-2f4a7ffa9636","Type":"ContainerStarted","Data":"c44c8b75eb6e91624c51641a44b7ad75d7e3fb64c0a021b5d813faff7e61e8aa"}
Nov 03 18:13:20.903709 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:20.903721    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7347-efc448" event={"ID":"dce1aa3b-8168-4aad-b1d8-2f4a7ffa9636","Type":"ContainerStarted","Data":"5554bb9cd55f32744163f06b84ae0603884faec7c788d6eb6b57c4e03df075d3"}
Nov 03 18:13:20.951791 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:13:20.951746    5315 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7347-a6fd67\" is forbidden: User \"system:node:ip-10-0-128-204.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-204.us-west-2.compute.internal' and this object" podUID="6516e404-f788-49ce-82ac-d819701ce0cf" pod="opendatahub/test-curl-7347-a6fd67"
Nov 03 18:13:20.999093 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:13:20.998995    5315 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7347-e1b8ef\" is forbidden: User \"system:node:ip-10-0-128-204.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-204.us-west-2.compute.internal' and this object" podUID="bd8c47da-62ba-4d3d-b257-8acedccdce77" pod="opendatahub/test-curl-7347-e1b8ef"
Nov 03 18:13:21.107490 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:21.107425    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7347-efc448" podStartSLOduration=2.107408121 podStartE2EDuration="2.107408121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:13:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:13:21.10458889 +0000 UTC m=+1640.120786963" watchObservedRunningTime="2026-11-03 18:13:21.107408121 +0000 UTC m=+1640.123606191"
Nov 03 18:13:21.111645 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:21.111599    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6516e404-f788-49ce-82ac-d819701ce0cf" path="/var/lib/kubelet/pods/6516e404-f788-49ce-82ac-d819701ce0cf/volumes"
Nov 03 18:13:21.207175 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:13:21.207131    5315 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7347-a6fd67\" is forbidden: User \"system:node:ip-10-0-128-204.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-204.us-west-2.compute.internal' and this object" podUID="6516e404-f788-49ce-82ac-d819701ce0cf" pod="opendatahub/test-curl-7347-a6fd67"
Nov 03 18:13:21.237580 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:21.237499    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b6cb8403-v7kr7"]
Nov 03 18:13:21.240946 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:13:21.240914    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45a6b9f7_0d90_448f_a209_5f5517530e72.slice/crio-93f151c067741ed0556e7a2a15bf03ca30a8231d01d06f4eee7f84e536ddf60a WatchSource:0}: Error finding container 93f151c067741ed0556e7a2a15bf03ca30a8231d01d06f4eee7f84e536ddf60a: Status 404 returned error can't find the container with id 93f151c067741ed0556e7a2a15bf03ca30a8231d01d06f4eee7f84e536ddf60a
Nov 03 18:13:21.254535 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:13:21.254476    5315 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7347-e1b8ef\" is forbidden: User \"system:node:ip-10-0-128-204.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-204.us-west-2.compute.internal' and this object" podUID="bd8c47da-62ba-4d3d-b257-8acedccdce77" pod="opendatahub/test-curl-7347-e1b8ef"
Nov 03 18:13:21.914366 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:21.914328    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b6cb8403-v7kr7" event={"ID":"45a6b9f7-0d90-448f-a209-5f5517530e72","Type":"ContainerStarted","Data":"0ef799abf00fafd0cd7246531d1810eb64a05b0e118cc960bc857b03e9059a44"}
Nov 03 18:13:21.914824 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:21.914395    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b6cb8403-v7kr7" event={"ID":"45a6b9f7-0d90-448f-a209-5f5517530e72","Type":"ContainerStarted","Data":"93f151c067741ed0556e7a2a15bf03ca30a8231d01d06f4eee7f84e536ddf60a"}
Nov 03 18:13:22.022556 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:22.022501    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b6cb8403-v7kr7" podStartSLOduration=2.022486761 podStartE2EDuration="2.022486761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:13:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:13:22.021309047 +0000 UTC m=+1641.037507117" watchObservedRunningTime="2026-11-03 18:13:22.022486761 +0000 UTC m=+1641.038684868"
Nov 03 18:13:22.158470 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:22.158400    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7347-e1b8ef"
Nov 03 18:13:22.234241 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:22.234207    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd8c47da-62ba-4d3d-b257-8acedccdce77-kube-api-access-brfvz\" (UniqueName: \"kubernetes.io/projected/bd8c47da-62ba-4d3d-b257-8acedccdce77-kube-api-access-brfvz\") pod \"bd8c47da-62ba-4d3d-b257-8acedccdce77\" (UID: \"bd8c47da-62ba-4d3d-b257-8acedccdce77\") "
Nov 03 18:13:22.236203 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:22.236177    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd8c47da-62ba-4d3d-b257-8acedccdce77-kube-api-access-brfvz" pod "bd8c47da-62ba-4d3d-b257-8acedccdce77" (UID: "bd8c47da-62ba-4d3d-b257-8acedccdce77"). InnerVolumeSpecName "kube-api-access-brfvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:13:22.336418 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:22.336380    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brfvz\" (UniqueName: \"kubernetes.io/projected/bd8c47da-62ba-4d3d-b257-8acedccdce77-kube-api-access-brfvz\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:22.547619 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:22.547571    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7347-efc448"]
Nov 03 18:13:22.551935 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:22.551906    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7347-efc448"]
Nov 03 18:13:22.724196 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:22.724164    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7347-d8b81e"]
Nov 03 18:13:22.724993 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:22.724971    5315 cpu_manager.go:414] "containerMap: removing container" podUID="dce1aa3b-8168-4aad-b1d8-2f4a7ffa9636" containerName="test-curl-7347-efc448"
Nov 03 18:13:22.725116 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:22.724993    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dce1aa3b-8168-4aad-b1d8-2f4a7ffa9636" containerName="test-curl-7347-efc448"
Nov 03 18:13:22.725173 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:22.725162    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="dce1aa3b-8168-4aad-b1d8-2f4a7ffa9636" containerName="test-curl-7347-efc448"
Nov 03 18:13:22.730305 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:22.730284    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7347-d8b81e"
Nov 03 18:13:22.740938 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:22.740909    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhqnt\" (UniqueName: \"kubernetes.io/projected/09c0ab7f-d68a-4322-a716-4ef244c53fd0-kube-api-access-mhqnt\") pod \"test-curl-7347-d8b81e\" (UID: \"09c0ab7f-d68a-4322-a716-4ef244c53fd0\") " pod="opendatahub/test-curl-7347-d8b81e"
Nov 03 18:13:22.784585 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:22.784551    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7347-d8b81e"]
Nov 03 18:13:22.842479 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:22.842383    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhqnt\" (UniqueName: \"kubernetes.io/projected/09c0ab7f-d68a-4322-a716-4ef244c53fd0-kube-api-access-mhqnt\") pod \"test-curl-7347-d8b81e\" (UID: \"09c0ab7f-d68a-4322-a716-4ef244c53fd0\") " pod="opendatahub/test-curl-7347-d8b81e"
Nov 03 18:13:22.899659 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:22.899624    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhqnt\" (UniqueName: \"kubernetes.io/projected/09c0ab7f-d68a-4322-a716-4ef244c53fd0-kube-api-access-mhqnt\") pod \"test-curl-7347-d8b81e\" (UID: \"09c0ab7f-d68a-4322-a716-4ef244c53fd0\") " pod="opendatahub/test-curl-7347-d8b81e"
Nov 03 18:13:22.924058 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:22.924024    5315 generic.go:356] "Generic (PLEG): container finished" podID="bd8c47da-62ba-4d3d-b257-8acedccdce77" containerID="5436a2b19e9c222279555aed1559bc0afea5a2d7c871918723c7d7f44cb71bd4" exitCode=137
Nov 03 18:13:22.924586 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:22.924065    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7347-e1b8ef"
Nov 03 18:13:22.924586 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:22.924117    5315 scope.go:122] "RemoveContainer" containerID="5436a2b19e9c222279555aed1559bc0afea5a2d7c871918723c7d7f44cb71bd4"
Nov 03 18:13:22.926148 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:22.926125    5315 generic.go:356] "Generic (PLEG): container finished" podID="45a6b9f7-0d90-448f-a209-5f5517530e72" containerID="0ef799abf00fafd0cd7246531d1810eb64a05b0e118cc960bc857b03e9059a44" exitCode=0
Nov 03 18:13:22.926406 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:22.926387    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7347-efc448" podUID="dce1aa3b-8168-4aad-b1d8-2f4a7ffa9636" containerName="test-curl-7347-efc448" containerID="cri-o://c44c8b75eb6e91624c51641a44b7ad75d7e3fb64c0a021b5d813faff7e61e8aa" gracePeriod=2
Nov 03 18:13:22.926800 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:22.926776    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b6cb8403-v7kr7" event={"ID":"45a6b9f7-0d90-448f-a209-5f5517530e72","Type":"ContainerDied","Data":"0ef799abf00fafd0cd7246531d1810eb64a05b0e118cc960bc857b03e9059a44"}
Nov 03 18:13:22.936166 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:22.936145    5315 scope.go:122] "RemoveContainer" containerID="5436a2b19e9c222279555aed1559bc0afea5a2d7c871918723c7d7f44cb71bd4"
Nov 03 18:13:22.936490 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:13:22.936473    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5436a2b19e9c222279555aed1559bc0afea5a2d7c871918723c7d7f44cb71bd4\": container with ID starting with 5436a2b19e9c222279555aed1559bc0afea5a2d7c871918723c7d7f44cb71bd4 not found: ID does not exist" containerID="5436a2b19e9c222279555aed1559bc0afea5a2d7c871918723c7d7f44cb71bd4"
Nov 03 18:13:22.936545 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:22.936497    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5436a2b19e9c222279555aed1559bc0afea5a2d7c871918723c7d7f44cb71bd4"} err="failed to get container status \"5436a2b19e9c222279555aed1559bc0afea5a2d7c871918723c7d7f44cb71bd4\": rpc error: code = NotFound desc = could not find container \"5436a2b19e9c222279555aed1559bc0afea5a2d7c871918723c7d7f44cb71bd4\": container with ID starting with 5436a2b19e9c222279555aed1559bc0afea5a2d7c871918723c7d7f44cb71bd4 not found: ID does not exist"
Nov 03 18:13:23.042528 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:23.042493    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7347-d8b81e"
Nov 03 18:13:23.083942 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:13:23.083888    5315 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7347-efc448\" is forbidden: User \"system:node:ip-10-0-128-204.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-204.us-west-2.compute.internal' and this object" podUID="dce1aa3b-8168-4aad-b1d8-2f4a7ffa9636" pod="opendatahub/test-curl-7347-efc448"
Nov 03 18:13:23.109321 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:23.109291    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd8c47da-62ba-4d3d-b257-8acedccdce77" path="/var/lib/kubelet/pods/bd8c47da-62ba-4d3d-b257-8acedccdce77/volumes"
Nov 03 18:13:23.131499 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:13:23.131456    5315 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7347-efc448\" is forbidden: User \"system:node:ip-10-0-128-204.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-204.us-west-2.compute.internal' and this object" podUID="dce1aa3b-8168-4aad-b1d8-2f4a7ffa9636" pod="opendatahub/test-curl-7347-efc448"
Nov 03 18:13:23.178980 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:13:23.178930    5315 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7347-e1b8ef\" is forbidden: User \"system:node:ip-10-0-128-204.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-204.us-west-2.compute.internal' and this object" podUID="bd8c47da-62ba-4d3d-b257-8acedccdce77" pod="opendatahub/test-curl-7347-e1b8ef"
Nov 03 18:13:23.421436 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:23.421334    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7347-d8b81e"]
Nov 03 18:13:23.426893 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:13:23.426858    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09c0ab7f_d68a_4322_a716_4ef244c53fd0.slice/crio-996d3e49f19b0e31539581a2a560ca3919cc126025511172fcf0be67bd5c97a4 WatchSource:0}: Error finding container 996d3e49f19b0e31539581a2a560ca3919cc126025511172fcf0be67bd5c97a4: Status 404 returned error can't find the container with id 996d3e49f19b0e31539581a2a560ca3919cc126025511172fcf0be67bd5c97a4
Nov 03 18:13:23.936412 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:23.936376    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7347-d8b81e" event={"ID":"09c0ab7f-d68a-4322-a716-4ef244c53fd0","Type":"ContainerStarted","Data":"134e17b0747a01284f2dfd4a926e4317aa9e4e56f7a2817113e74b37eb198eb0"}
Nov 03 18:13:23.936412 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:23.936418    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7347-d8b81e" event={"ID":"09c0ab7f-d68a-4322-a716-4ef244c53fd0","Type":"ContainerStarted","Data":"996d3e49f19b0e31539581a2a560ca3919cc126025511172fcf0be67bd5c97a4"}
Nov 03 18:13:24.047085 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:24.047022    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7347-d8b81e" podStartSLOduration=2.046999553 podStartE2EDuration="2.046999553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:13:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:13:24.045496621 +0000 UTC m=+1643.061694697" watchObservedRunningTime="2026-11-03 18:13:24.046999553 +0000 UTC m=+1643.063197622"
Nov 03 18:13:24.136548 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:24.136517    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b6cb8403-v7kr7"
Nov 03 18:13:24.160876 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:24.160846    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45a6b9f7-0d90-448f-a209-5f5517530e72-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45a6b9f7-0d90-448f-a209-5f5517530e72-maas-api-xks-ca\") pod \"45a6b9f7-0d90-448f-a209-5f5517530e72\" (UID: \"45a6b9f7-0d90-448f-a209-5f5517530e72\") "
Nov 03 18:13:24.161040 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:24.161028    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45a6b9f7-0d90-448f-a209-5f5517530e72-kube-api-access-8r5fr\" (UniqueName: \"kubernetes.io/projected/45a6b9f7-0d90-448f-a209-5f5517530e72-kube-api-access-8r5fr\") pod \"45a6b9f7-0d90-448f-a209-5f5517530e72\" (UID: \"45a6b9f7-0d90-448f-a209-5f5517530e72\") "
Nov 03 18:13:24.161082 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:24.161055    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45a6b9f7-0d90-448f-a209-5f5517530e72-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45a6b9f7-0d90-448f-a209-5f5517530e72-maas-api-openshift-ca\") pod \"45a6b9f7-0d90-448f-a209-5f5517530e72\" (UID: \"45a6b9f7-0d90-448f-a209-5f5517530e72\") "
Nov 03 18:13:24.161453 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:24.161414    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45a6b9f7-0d90-448f-a209-5f5517530e72-maas-api-openshift-ca" pod "45a6b9f7-0d90-448f-a209-5f5517530e72" (UID: "45a6b9f7-0d90-448f-a209-5f5517530e72"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:13:24.161648 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:24.161623    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45a6b9f7-0d90-448f-a209-5f5517530e72-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:24.163122 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:24.163098    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45a6b9f7-0d90-448f-a209-5f5517530e72-maas-api-xks-ca" pod "45a6b9f7-0d90-448f-a209-5f5517530e72" (UID: "45a6b9f7-0d90-448f-a209-5f5517530e72"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:13:24.163230 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:24.163175    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45a6b9f7-0d90-448f-a209-5f5517530e72-kube-api-access-8r5fr" pod "45a6b9f7-0d90-448f-a209-5f5517530e72" (UID: "45a6b9f7-0d90-448f-a209-5f5517530e72"). InnerVolumeSpecName "kube-api-access-8r5fr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:13:24.262817 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:24.262744    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r5fr\" (UniqueName: \"kubernetes.io/projected/45a6b9f7-0d90-448f-a209-5f5517530e72-kube-api-access-8r5fr\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:24.262817 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:24.262774    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45a6b9f7-0d90-448f-a209-5f5517530e72-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:24.820195 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:13:24.820117    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:13:25.070909 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:25.063647    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b6cb8403-v7kr7"
Nov 03 18:13:25.070909 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:25.069851    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b6cb8403-v7kr7" event={"ID":"45a6b9f7-0d90-448f-a209-5f5517530e72","Type":"ContainerDied","Data":"93f151c067741ed0556e7a2a15bf03ca30a8231d01d06f4eee7f84e536ddf60a"}
Nov 03 18:13:25.070909 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:25.069926    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93f151c067741ed0556e7a2a15bf03ca30a8231d01d06f4eee7f84e536ddf60a"
Nov 03 18:13:25.360298 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:25.360259    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7347-efc448"
Nov 03 18:13:25.411052 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:25.411022    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dce1aa3b-8168-4aad-b1d8-2f4a7ffa9636-kube-api-access-djdkb\" (UniqueName: \"kubernetes.io/projected/dce1aa3b-8168-4aad-b1d8-2f4a7ffa9636-kube-api-access-djdkb\") pod \"dce1aa3b-8168-4aad-b1d8-2f4a7ffa9636\" (UID: \"dce1aa3b-8168-4aad-b1d8-2f4a7ffa9636\") "
Nov 03 18:13:25.413431 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:25.413397    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dce1aa3b-8168-4aad-b1d8-2f4a7ffa9636-kube-api-access-djdkb" pod "dce1aa3b-8168-4aad-b1d8-2f4a7ffa9636" (UID: "dce1aa3b-8168-4aad-b1d8-2f4a7ffa9636"). InnerVolumeSpecName "kube-api-access-djdkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:13:25.513257 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:25.513221    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djdkb\" (UniqueName: \"kubernetes.io/projected/dce1aa3b-8168-4aad-b1d8-2f4a7ffa9636-kube-api-access-djdkb\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:26.001634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:25.992666    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7347-d8b81e"]
Nov 03 18:13:26.014633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:26.007878    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7347-d8b81e"]
Nov 03 18:13:26.175660 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:26.174909    5315 generic.go:356] "Generic (PLEG): container finished" podID="dce1aa3b-8168-4aad-b1d8-2f4a7ffa9636" containerID="c44c8b75eb6e91624c51641a44b7ad75d7e3fb64c0a021b5d813faff7e61e8aa" exitCode=137
Nov 03 18:13:26.175660 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:26.175176    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7347-d8b81e" podUID="09c0ab7f-d68a-4322-a716-4ef244c53fd0" containerName="test-curl-7347-d8b81e" containerID="cri-o://134e17b0747a01284f2dfd4a926e4317aa9e4e56f7a2817113e74b37eb198eb0" gracePeriod=2
Nov 03 18:13:26.175660 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:26.175569    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7347-efc448"
Nov 03 18:13:26.178281 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:26.178004    5315 scope.go:122] "RemoveContainer" containerID="c44c8b75eb6e91624c51641a44b7ad75d7e3fb64c0a021b5d813faff7e61e8aa"
Nov 03 18:13:26.200558 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:26.200534    5315 scope.go:122] "RemoveContainer" containerID="c44c8b75eb6e91624c51641a44b7ad75d7e3fb64c0a021b5d813faff7e61e8aa"
Nov 03 18:13:26.200983 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:13:26.200960    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c44c8b75eb6e91624c51641a44b7ad75d7e3fb64c0a021b5d813faff7e61e8aa\": container with ID starting with c44c8b75eb6e91624c51641a44b7ad75d7e3fb64c0a021b5d813faff7e61e8aa not found: ID does not exist" containerID="c44c8b75eb6e91624c51641a44b7ad75d7e3fb64c0a021b5d813faff7e61e8aa"
Nov 03 18:13:26.201069 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:26.200995    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c44c8b75eb6e91624c51641a44b7ad75d7e3fb64c0a021b5d813faff7e61e8aa"} err="failed to get container status \"c44c8b75eb6e91624c51641a44b7ad75d7e3fb64c0a021b5d813faff7e61e8aa\": rpc error: code = NotFound desc = could not find container \"c44c8b75eb6e91624c51641a44b7ad75d7e3fb64c0a021b5d813faff7e61e8aa\": container with ID starting with c44c8b75eb6e91624c51641a44b7ad75d7e3fb64c0a021b5d813faff7e61e8aa not found: ID does not exist"
Nov 03 18:13:26.225054 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:13:26.225013    5315 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7347-d8b81e\" is forbidden: User \"system:node:ip-10-0-128-204.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-204.us-west-2.compute.internal' and this object" podUID="09c0ab7f-d68a-4322-a716-4ef244c53fd0" pod="opendatahub/test-curl-7347-d8b81e"
Nov 03 18:13:26.272472 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:13:26.272372    5315 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7347-d8b81e\" is forbidden: User \"system:node:ip-10-0-128-204.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-204.us-west-2.compute.internal' and this object" podUID="09c0ab7f-d68a-4322-a716-4ef244c53fd0" pod="opendatahub/test-curl-7347-d8b81e"
Nov 03 18:13:26.319627 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:13:26.319580    5315 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7347-efc448\" is forbidden: User \"system:node:ip-10-0-128-204.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-204.us-west-2.compute.internal' and this object" podUID="dce1aa3b-8168-4aad-b1d8-2f4a7ffa9636" pod="opendatahub/test-curl-7347-efc448"
Nov 03 18:13:26.563633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:26.562925    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c91bee-5vfdm"]
Nov 03 18:13:26.566647 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:26.564097    5315 cpu_manager.go:414] "containerMap: removing container" podUID="45a6b9f7-0d90-448f-a209-5f5517530e72" containerName="revoke-keys"
Nov 03 18:13:26.566647 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:26.564126    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45a6b9f7-0d90-448f-a209-5f5517530e72" containerName="revoke-keys"
Nov 03 18:13:26.566647 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:26.564163    5315 cpu_manager.go:414] "containerMap: removing container" podUID="09c0ab7f-d68a-4322-a716-4ef244c53fd0" containerName="test-curl-7347-d8b81e"
Nov 03 18:13:26.566647 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:26.564171    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09c0ab7f-d68a-4322-a716-4ef244c53fd0" containerName="test-curl-7347-d8b81e"
Nov 03 18:13:26.566647 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:26.564414    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="45a6b9f7-0d90-448f-a209-5f5517530e72" containerName="revoke-keys"
Nov 03 18:13:26.566647 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:26.564436    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="09c0ab7f-d68a-4322-a716-4ef244c53fd0" containerName="test-curl-7347-d8b81e"
Nov 03 18:13:26.572667 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:26.572028    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c91bee-5vfdm"
Nov 03 18:13:26.599659 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:26.599128    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c91bee-5vfdm"]
Nov 03 18:13:26.625882 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:26.625795    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:13:26.625882 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:26.625850    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:13:26.626167 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:26.625923    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:13:26.626785 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:26.626742    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:13:26.655225 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:26.654700    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9569cd4c-0762-444e-88de-8e6825323878-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-c91bee-5vfdm\" (UID: \"9569cd4c-0762-444e-88de-8e6825323878\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c91bee-5vfdm"
Nov 03 18:13:26.655225 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:26.654786    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j68v\" (UniqueName: \"kubernetes.io/projected/9569cd4c-0762-444e-88de-8e6825323878-kube-api-access-7j68v\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-c91bee-5vfdm\" (UID: \"9569cd4c-0762-444e-88de-8e6825323878\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c91bee-5vfdm"
Nov 03 18:13:26.655225 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:26.655001    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9569cd4c-0762-444e-88de-8e6825323878-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-c91bee-5vfdm\" (UID: \"9569cd4c-0762-444e-88de-8e6825323878\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c91bee-5vfdm"
Nov 03 18:13:26.737662 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:13:26.737591    5315 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7347-d8b81e\" is forbidden: User \"system:node:ip-10-0-128-204.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-204.us-west-2.compute.internal' and this object" podUID="09c0ab7f-d68a-4322-a716-4ef244c53fd0" pod="opendatahub/test-curl-7347-d8b81e"
Nov 03 18:13:26.767391 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:26.759934    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9569cd4c-0762-444e-88de-8e6825323878-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-c91bee-5vfdm\" (UID: \"9569cd4c-0762-444e-88de-8e6825323878\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c91bee-5vfdm"
Nov 03 18:13:26.767890 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:26.767860    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9569cd4c-0762-444e-88de-8e6825323878-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-c91bee-5vfdm\" (UID: \"9569cd4c-0762-444e-88de-8e6825323878\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c91bee-5vfdm"
Nov 03 18:13:26.768075 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:26.768059    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j68v\" (UniqueName: \"kubernetes.io/projected/9569cd4c-0762-444e-88de-8e6825323878-kube-api-access-7j68v\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-c91bee-5vfdm\" (UID: \"9569cd4c-0762-444e-88de-8e6825323878\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c91bee-5vfdm"
Nov 03 18:13:26.769532 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:26.769504    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9569cd4c-0762-444e-88de-8e6825323878-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-c91bee-5vfdm\" (UID: \"9569cd4c-0762-444e-88de-8e6825323878\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c91bee-5vfdm"
Nov 03 18:13:26.771499 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:26.771474    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9569cd4c-0762-444e-88de-8e6825323878-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-c91bee-5vfdm\" (UID: \"9569cd4c-0762-444e-88de-8e6825323878\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c91bee-5vfdm"
Nov 03 18:13:26.786156 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:13:26.786117    5315 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7347-efc448\" is forbidden: User \"system:node:ip-10-0-128-204.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-204.us-west-2.compute.internal' and this object" podUID="dce1aa3b-8168-4aad-b1d8-2f4a7ffa9636" pod="opendatahub/test-curl-7347-efc448"
Nov 03 18:13:26.825389 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:26.825312    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j68v\" (UniqueName: \"kubernetes.io/projected/9569cd4c-0762-444e-88de-8e6825323878-kube-api-access-7j68v\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-c91bee-5vfdm\" (UID: \"9569cd4c-0762-444e-88de-8e6825323878\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c91bee-5vfdm"
Nov 03 18:13:26.921303 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:26.921266    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c91bee-5vfdm"
Nov 03 18:13:27.105645 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:27.105550    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dce1aa3b-8168-4aad-b1d8-2f4a7ffa9636" path="/var/lib/kubelet/pods/dce1aa3b-8168-4aad-b1d8-2f4a7ffa9636/volumes"
Nov 03 18:13:27.314239 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:27.314205    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c91bee-5vfdm"]
Nov 03 18:13:27.319718 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:13:27.319690    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9569cd4c_0762_444e_88de_8e6825323878.slice/crio-d8f5f1d7b33f1426e641b9a30a3f52d7679131e6fa2e7e9a334ef55c58974cb0 WatchSource:0}: Error finding container d8f5f1d7b33f1426e641b9a30a3f52d7679131e6fa2e7e9a334ef55c58974cb0: Status 404 returned error can't find the container with id d8f5f1d7b33f1426e641b9a30a3f52d7679131e6fa2e7e9a334ef55c58974cb0
Nov 03 18:13:28.195247 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:28.195211    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c91bee-5vfdm" event={"ID":"9569cd4c-0762-444e-88de-8e6825323878","Type":"ContainerStarted","Data":"eed7385d3e244bb7b2aa09105e3dfd35ae02112d466bf21436ed51ce52a4edcc"}
Nov 03 18:13:28.195400 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:28.195277    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c91bee-5vfdm" event={"ID":"9569cd4c-0762-444e-88de-8e6825323878","Type":"ContainerStarted","Data":"d8f5f1d7b33f1426e641b9a30a3f52d7679131e6fa2e7e9a334ef55c58974cb0"}
Nov 03 18:13:28.306323 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:28.306264    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c91bee-5vfdm" podStartSLOduration=2.306249226 podStartE2EDuration="2.306249226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:13:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:13:28.302985327 +0000 UTC m=+1647.319183387" watchObservedRunningTime="2026-11-03 18:13:28.306249226 +0000 UTC m=+1647.322447295"
Nov 03 18:13:28.516090 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:28.516059    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7347-d8b81e"
Nov 03 18:13:28.706154 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:28.705559    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09c0ab7f-d68a-4322-a716-4ef244c53fd0-kube-api-access-mhqnt\" (UniqueName: \"kubernetes.io/projected/09c0ab7f-d68a-4322-a716-4ef244c53fd0-kube-api-access-mhqnt\") pod \"09c0ab7f-d68a-4322-a716-4ef244c53fd0\" (UID: \"09c0ab7f-d68a-4322-a716-4ef244c53fd0\") "
Nov 03 18:13:28.708243 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:28.708216    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09c0ab7f-d68a-4322-a716-4ef244c53fd0-kube-api-access-mhqnt" pod "09c0ab7f-d68a-4322-a716-4ef244c53fd0" (UID: "09c0ab7f-d68a-4322-a716-4ef244c53fd0"). InnerVolumeSpecName "kube-api-access-mhqnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:13:28.807153 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:28.807116    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhqnt\" (UniqueName: \"kubernetes.io/projected/09c0ab7f-d68a-4322-a716-4ef244c53fd0-kube-api-access-mhqnt\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:29.106364 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:29.106283    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09c0ab7f-d68a-4322-a716-4ef244c53fd0" path="/var/lib/kubelet/pods/09c0ab7f-d68a-4322-a716-4ef244c53fd0/volumes"
Nov 03 18:13:29.204185 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:29.204148    5315 generic.go:356] "Generic (PLEG): container finished" podID="9569cd4c-0762-444e-88de-8e6825323878" containerID="eed7385d3e244bb7b2aa09105e3dfd35ae02112d466bf21436ed51ce52a4edcc" exitCode=0
Nov 03 18:13:29.204360 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:29.204232    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c91bee-5vfdm" event={"ID":"9569cd4c-0762-444e-88de-8e6825323878","Type":"ContainerDied","Data":"eed7385d3e244bb7b2aa09105e3dfd35ae02112d466bf21436ed51ce52a4edcc"}
Nov 03 18:13:29.206713 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:29.206681    5315 generic.go:356] "Generic (PLEG): container finished" podID="09c0ab7f-d68a-4322-a716-4ef244c53fd0" containerID="134e17b0747a01284f2dfd4a926e4317aa9e4e56f7a2817113e74b37eb198eb0" exitCode=137
Nov 03 18:13:29.206853 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:29.206720    5315 scope.go:122] "RemoveContainer" containerID="134e17b0747a01284f2dfd4a926e4317aa9e4e56f7a2817113e74b37eb198eb0"
Nov 03 18:13:29.206853 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:29.206736    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7347-d8b81e"
Nov 03 18:13:29.217712 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:29.217689    5315 scope.go:122] "RemoveContainer" containerID="134e17b0747a01284f2dfd4a926e4317aa9e4e56f7a2817113e74b37eb198eb0"
Nov 03 18:13:29.218034 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:13:29.218014    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"134e17b0747a01284f2dfd4a926e4317aa9e4e56f7a2817113e74b37eb198eb0\": container with ID starting with 134e17b0747a01284f2dfd4a926e4317aa9e4e56f7a2817113e74b37eb198eb0 not found: ID does not exist" containerID="134e17b0747a01284f2dfd4a926e4317aa9e4e56f7a2817113e74b37eb198eb0"
Nov 03 18:13:29.218096 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:29.218043    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"134e17b0747a01284f2dfd4a926e4317aa9e4e56f7a2817113e74b37eb198eb0"} err="failed to get container status \"134e17b0747a01284f2dfd4a926e4317aa9e4e56f7a2817113e74b37eb198eb0\": rpc error: code = NotFound desc = could not find container \"134e17b0747a01284f2dfd4a926e4317aa9e4e56f7a2817113e74b37eb198eb0\": container with ID starting with 134e17b0747a01284f2dfd4a926e4317aa9e4e56f7a2817113e74b37eb198eb0 not found: ID does not exist"
Nov 03 18:13:29.359694 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:29.355801    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0ccbb9-7tm97"]
Nov 03 18:13:29.374689 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:29.374474    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0ccbb9-7tm97"
Nov 03 18:13:29.416344 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:29.416302    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwhjd\" (UniqueName: \"kubernetes.io/projected/ad347a4e-f710-4482-a326-5e05d12b0dc3-kube-api-access-hwhjd\") pod \"maas-api-revoke-keys-e2e-worker-w0-0ccbb9-7tm97\" (UID: \"ad347a4e-f710-4482-a326-5e05d12b0dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0ccbb9-7tm97"
Nov 03 18:13:29.416501 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:29.416470    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad347a4e-f710-4482-a326-5e05d12b0dc3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-0ccbb9-7tm97\" (UID: \"ad347a4e-f710-4482-a326-5e05d12b0dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0ccbb9-7tm97"
Nov 03 18:13:29.416599 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:29.416579    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad347a4e-f710-4482-a326-5e05d12b0dc3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-0ccbb9-7tm97\" (UID: \"ad347a4e-f710-4482-a326-5e05d12b0dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0ccbb9-7tm97"
Nov 03 18:13:29.480634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:29.478661    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0ccbb9-7tm97"]
Nov 03 18:13:29.518672 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:29.517983    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwhjd\" (UniqueName: \"kubernetes.io/projected/ad347a4e-f710-4482-a326-5e05d12b0dc3-kube-api-access-hwhjd\") pod \"maas-api-revoke-keys-e2e-worker-w0-0ccbb9-7tm97\" (UID: \"ad347a4e-f710-4482-a326-5e05d12b0dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0ccbb9-7tm97"
Nov 03 18:13:29.518672 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:29.518145    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad347a4e-f710-4482-a326-5e05d12b0dc3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-0ccbb9-7tm97\" (UID: \"ad347a4e-f710-4482-a326-5e05d12b0dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0ccbb9-7tm97"
Nov 03 18:13:29.518672 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:29.518234    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad347a4e-f710-4482-a326-5e05d12b0dc3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-0ccbb9-7tm97\" (UID: \"ad347a4e-f710-4482-a326-5e05d12b0dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0ccbb9-7tm97"
Nov 03 18:13:29.519280 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:29.518843    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad347a4e-f710-4482-a326-5e05d12b0dc3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-0ccbb9-7tm97\" (UID: \"ad347a4e-f710-4482-a326-5e05d12b0dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0ccbb9-7tm97"
Nov 03 18:13:29.521367 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:29.521336    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad347a4e-f710-4482-a326-5e05d12b0dc3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-0ccbb9-7tm97\" (UID: \"ad347a4e-f710-4482-a326-5e05d12b0dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0ccbb9-7tm97"
Nov 03 18:13:29.579685 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:29.579645    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwhjd\" (UniqueName: \"kubernetes.io/projected/ad347a4e-f710-4482-a326-5e05d12b0dc3-kube-api-access-hwhjd\") pod \"maas-api-revoke-keys-e2e-worker-w0-0ccbb9-7tm97\" (UID: \"ad347a4e-f710-4482-a326-5e05d12b0dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0ccbb9-7tm97"
Nov 03 18:13:29.694788 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:29.694708    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0ccbb9-7tm97"
Nov 03 18:13:30.182829 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:30.180978    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0ccbb9-7tm97"]
Nov 03 18:13:30.183296 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:13:30.183267    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad347a4e_f710_4482_a326_5e05d12b0dc3.slice/crio-0d4477ba6cb8d8d1ff321785b4a419fc0ea27e33658f977af3b4303eb0f9d787 WatchSource:0}: Error finding container 0d4477ba6cb8d8d1ff321785b4a419fc0ea27e33658f977af3b4303eb0f9d787: Status 404 returned error can't find the container with id 0d4477ba6cb8d8d1ff321785b4a419fc0ea27e33658f977af3b4303eb0f9d787
Nov 03 18:13:30.217411 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:30.217379    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0ccbb9-7tm97" event={"ID":"ad347a4e-f710-4482-a326-5e05d12b0dc3","Type":"ContainerStarted","Data":"0d4477ba6cb8d8d1ff321785b4a419fc0ea27e33658f977af3b4303eb0f9d787"}
Nov 03 18:13:30.604631 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:30.604588    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c91bee-5vfdm"
Nov 03 18:13:30.692333 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:30.692299    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9569cd4c-0762-444e-88de-8e6825323878-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9569cd4c-0762-444e-88de-8e6825323878-maas-api-openshift-ca\") pod \"9569cd4c-0762-444e-88de-8e6825323878\" (UID: \"9569cd4c-0762-444e-88de-8e6825323878\") "
Nov 03 18:13:30.692549 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:30.692534    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9569cd4c-0762-444e-88de-8e6825323878-kube-api-access-7j68v\" (UniqueName: \"kubernetes.io/projected/9569cd4c-0762-444e-88de-8e6825323878-kube-api-access-7j68v\") pod \"9569cd4c-0762-444e-88de-8e6825323878\" (UID: \"9569cd4c-0762-444e-88de-8e6825323878\") "
Nov 03 18:13:30.692596 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:30.692583    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9569cd4c-0762-444e-88de-8e6825323878-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9569cd4c-0762-444e-88de-8e6825323878-maas-api-xks-ca\") pod \"9569cd4c-0762-444e-88de-8e6825323878\" (UID: \"9569cd4c-0762-444e-88de-8e6825323878\") "
Nov 03 18:13:30.693161 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:30.693137    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9569cd4c-0762-444e-88de-8e6825323878-maas-api-openshift-ca" pod "9569cd4c-0762-444e-88de-8e6825323878" (UID: "9569cd4c-0762-444e-88de-8e6825323878"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:13:30.695152 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:30.695123    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9569cd4c-0762-444e-88de-8e6825323878-maas-api-xks-ca" pod "9569cd4c-0762-444e-88de-8e6825323878" (UID: "9569cd4c-0762-444e-88de-8e6825323878"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:13:30.695152 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:30.695134    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9569cd4c-0762-444e-88de-8e6825323878-kube-api-access-7j68v" pod "9569cd4c-0762-444e-88de-8e6825323878" (UID: "9569cd4c-0762-444e-88de-8e6825323878"). InnerVolumeSpecName "kube-api-access-7j68v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:13:30.794629 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:30.794519    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j68v\" (UniqueName: \"kubernetes.io/projected/9569cd4c-0762-444e-88de-8e6825323878-kube-api-access-7j68v\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:30.794629 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:30.794559    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9569cd4c-0762-444e-88de-8e6825323878-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:30.794629 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:30.794573    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9569cd4c-0762-444e-88de-8e6825323878-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:31.281396 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:31.267817    5315 generic.go:356] "Generic (PLEG): container finished" podID="ad347a4e-f710-4482-a326-5e05d12b0dc3" containerID="cfd88ada69ad0317fe2bd1b799dc75b9d201d139841d4cb4dfc5bf06702e4c16" exitCode=0
Nov 03 18:13:31.281396 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:31.268027    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0ccbb9-7tm97" event={"ID":"ad347a4e-f710-4482-a326-5e05d12b0dc3","Type":"ContainerDied","Data":"cfd88ada69ad0317fe2bd1b799dc75b9d201d139841d4cb4dfc5bf06702e4c16"}
Nov 03 18:13:31.281396 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:31.271153    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c91bee-5vfdm" event={"ID":"9569cd4c-0762-444e-88de-8e6825323878","Type":"ContainerDied","Data":"d8f5f1d7b33f1426e641b9a30a3f52d7679131e6fa2e7e9a334ef55c58974cb0"}
Nov 03 18:13:31.281396 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:31.271208    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8f5f1d7b33f1426e641b9a30a3f52d7679131e6fa2e7e9a334ef55c58974cb0"
Nov 03 18:13:31.281396 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:31.271287    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c91bee-5vfdm"
Nov 03 18:13:32.586019 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:32.585986    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0ccbb9-7tm97"
Nov 03 18:13:32.678632 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:32.678574    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad347a4e-f710-4482-a326-5e05d12b0dc3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad347a4e-f710-4482-a326-5e05d12b0dc3-maas-api-openshift-ca\") pod \"ad347a4e-f710-4482-a326-5e05d12b0dc3\" (UID: \"ad347a4e-f710-4482-a326-5e05d12b0dc3\") "
Nov 03 18:13:32.678933 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:32.678910    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad347a4e-f710-4482-a326-5e05d12b0dc3-kube-api-access-hwhjd\" (UniqueName: \"kubernetes.io/projected/ad347a4e-f710-4482-a326-5e05d12b0dc3-kube-api-access-hwhjd\") pod \"ad347a4e-f710-4482-a326-5e05d12b0dc3\" (UID: \"ad347a4e-f710-4482-a326-5e05d12b0dc3\") "
Nov 03 18:13:32.679074 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:32.678945    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad347a4e-f710-4482-a326-5e05d12b0dc3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad347a4e-f710-4482-a326-5e05d12b0dc3-maas-api-xks-ca\") pod \"ad347a4e-f710-4482-a326-5e05d12b0dc3\" (UID: \"ad347a4e-f710-4482-a326-5e05d12b0dc3\") "
Nov 03 18:13:32.679074 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:32.678964    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad347a4e-f710-4482-a326-5e05d12b0dc3-maas-api-openshift-ca" pod "ad347a4e-f710-4482-a326-5e05d12b0dc3" (UID: "ad347a4e-f710-4482-a326-5e05d12b0dc3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:13:32.679411 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:32.679391    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad347a4e-f710-4482-a326-5e05d12b0dc3-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:32.681130 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:32.681099    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad347a4e-f710-4482-a326-5e05d12b0dc3-kube-api-access-hwhjd" pod "ad347a4e-f710-4482-a326-5e05d12b0dc3" (UID: "ad347a4e-f710-4482-a326-5e05d12b0dc3"). InnerVolumeSpecName "kube-api-access-hwhjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:13:32.681234 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:32.681153    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad347a4e-f710-4482-a326-5e05d12b0dc3-maas-api-xks-ca" pod "ad347a4e-f710-4482-a326-5e05d12b0dc3" (UID: "ad347a4e-f710-4482-a326-5e05d12b0dc3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:13:32.780732 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:32.780690    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwhjd\" (UniqueName: \"kubernetes.io/projected/ad347a4e-f710-4482-a326-5e05d12b0dc3-kube-api-access-hwhjd\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:32.780732 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:32.780732    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad347a4e-f710-4482-a326-5e05d12b0dc3-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:33.087706 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:33.087663    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b6cb8403-5545f5d6b7-6m5gp"]
Nov 03 18:13:33.210900 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:33.210861    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b6cb8403-77775ddc9f-pmzcv"]
Nov 03 18:13:33.296144 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:33.296067    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0ccbb9-7tm97"
Nov 03 18:13:33.296302 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:33.296072    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0ccbb9-7tm97" event={"ID":"ad347a4e-f710-4482-a326-5e05d12b0dc3","Type":"ContainerDied","Data":"0d4477ba6cb8d8d1ff321785b4a419fc0ea27e33658f977af3b4303eb0f9d787"}
Nov 03 18:13:33.296302 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:33.296194    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d4477ba6cb8d8d1ff321785b4a419fc0ea27e33658f977af3b4303eb0f9d787"
Nov 03 18:13:33.841470 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:33.841430    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s"]
Nov 03 18:13:38.088157 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:38.088099    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-b6cb8403-5545f5d6b7-6m5gp" podUID="581c90bc-60cd-412a-b4d5-5bbab2b49336" containerName="payload-processing" containerID="cri-o://40d1300622a48f81b2e1a83b63f4b1be759c524180ae9c09713410b6db8814b1" gracePeriod=25
Nov 03 18:13:38.535429 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:38.092476    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-c91bee-76b997676c-zfjvq"]
Nov 03 18:13:38.535429 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:38.211444    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-b6cb8403-77775ddc9f-pmzcv" podUID="25bc2947-958b-4181-99a7-716108abe703" containerName="payload-pre-processing" containerID="cri-o://24ed2687b798f034b1d198ae91be48585c7d6e433657c5c342fa984a27fe7d02" gracePeriod=25
Nov 03 18:13:38.535429 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:38.215903    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c91bee-687764d75f-5dlqr"]
Nov 03 18:13:38.535429 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:38.370598    5315 generic.go:356] "Generic (PLEG): container finished" podID="25bc2947-958b-4181-99a7-716108abe703" containerID="24ed2687b798f034b1d198ae91be48585c7d6e433657c5c342fa984a27fe7d02" exitCode=0
Nov 03 18:13:38.535429 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:38.370922    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b6cb8403-77775ddc9f-pmzcv" event={"ID":"25bc2947-958b-4181-99a7-716108abe703","Type":"ContainerDied","Data":"24ed2687b798f034b1d198ae91be48585c7d6e433657c5c342fa984a27fe7d02"}
Nov 03 18:13:38.535429 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:38.372703    5315 generic.go:356] "Generic (PLEG): container finished" podID="581c90bc-60cd-412a-b4d5-5bbab2b49336" containerID="40d1300622a48f81b2e1a83b63f4b1be759c524180ae9c09713410b6db8814b1" exitCode=0
Nov 03 18:13:38.535429 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:38.372968    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b6cb8403-5545f5d6b7-6m5gp" event={"ID":"581c90bc-60cd-412a-b4d5-5bbab2b49336","Type":"ContainerDied","Data":"40d1300622a48f81b2e1a83b63f4b1be759c524180ae9c09713410b6db8814b1"}
Nov 03 18:13:38.535429 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:38.438057    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b6cb8403-5545f5d6b7-6m5gp"
Nov 03 18:13:38.554180 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:38.547444    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/581c90bc-60cd-412a-b4d5-5bbab2b49336-kube-api-access-nnfr4\" (UniqueName: \"kubernetes.io/projected/581c90bc-60cd-412a-b4d5-5bbab2b49336-kube-api-access-nnfr4\") pod \"581c90bc-60cd-412a-b4d5-5bbab2b49336\" (UID: \"581c90bc-60cd-412a-b4d5-5bbab2b49336\") "
Nov 03 18:13:38.554180 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:38.547741    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/581c90bc-60cd-412a-b4d5-5bbab2b49336-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/581c90bc-60cd-412a-b4d5-5bbab2b49336-plugins-config-volume\") pod \"581c90bc-60cd-412a-b4d5-5bbab2b49336\" (UID: \"581c90bc-60cd-412a-b4d5-5bbab2b49336\") "
Nov 03 18:13:38.566943 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:38.566894    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/581c90bc-60cd-412a-b4d5-5bbab2b49336-plugins-config-volume" pod "581c90bc-60cd-412a-b4d5-5bbab2b49336" (UID: "581c90bc-60cd-412a-b4d5-5bbab2b49336"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:13:38.612790 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:38.612748    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/581c90bc-60cd-412a-b4d5-5bbab2b49336-kube-api-access-nnfr4" pod "581c90bc-60cd-412a-b4d5-5bbab2b49336" (UID: "581c90bc-60cd-412a-b4d5-5bbab2b49336"). InnerVolumeSpecName "kube-api-access-nnfr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:13:38.624911 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:38.624785    5315 patch_prober.go:29] interesting pod/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:13:38.625272 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:38.624875    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r" podUID="36d4f867-6750-43de-8cdc-76f7e76fdd44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:13:38.640578 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:38.640214    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b6cb8403-77775ddc9f-pmzcv"
Nov 03 18:13:38.651643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:38.651542    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnfr4\" (UniqueName: \"kubernetes.io/projected/581c90bc-60cd-412a-b4d5-5bbab2b49336-kube-api-access-nnfr4\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:38.651643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:38.651582    5315 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/581c90bc-60cd-412a-b4d5-5bbab2b49336-plugins-config-volume\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:38.808564 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:38.752245    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25bc2947-958b-4181-99a7-716108abe703-kube-api-access-ltqvh\" (UniqueName: \"kubernetes.io/projected/25bc2947-958b-4181-99a7-716108abe703-kube-api-access-ltqvh\") pod \"25bc2947-958b-4181-99a7-716108abe703\" (UID: \"25bc2947-958b-4181-99a7-716108abe703\") "
Nov 03 18:13:38.808564 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:38.785814    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25bc2947-958b-4181-99a7-716108abe703-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25bc2947-958b-4181-99a7-716108abe703-plugins-config-volume\") pod \"25bc2947-958b-4181-99a7-716108abe703\" (UID: \"25bc2947-958b-4181-99a7-716108abe703\") "
Nov 03 18:13:38.808564 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:38.799209    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25bc2947-958b-4181-99a7-716108abe703-kube-api-access-ltqvh" pod "25bc2947-958b-4181-99a7-716108abe703" (UID: "25bc2947-958b-4181-99a7-716108abe703"). InnerVolumeSpecName "kube-api-access-ltqvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:13:38.808564 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:38.808418    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25bc2947-958b-4181-99a7-716108abe703-plugins-config-volume" pod "25bc2947-958b-4181-99a7-716108abe703" (UID: "25bc2947-958b-4181-99a7-716108abe703"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:13:38.888682 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:38.888413    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltqvh\" (UniqueName: \"kubernetes.io/projected/25bc2947-958b-4181-99a7-716108abe703-kube-api-access-ltqvh\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:38.888682 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:38.888447    5315 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25bc2947-958b-4181-99a7-716108abe703-plugins-config-volume\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:39.382007 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:39.381915    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b6cb8403-5545f5d6b7-6m5gp"
Nov 03 18:13:39.382007 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:39.381927    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b6cb8403-5545f5d6b7-6m5gp" event={"ID":"581c90bc-60cd-412a-b4d5-5bbab2b49336","Type":"ContainerDied","Data":"e871229c62613882bf0b9148ed9bfb52f55c0cd697dd4abde9f97adf82d2ce00"}
Nov 03 18:13:39.382007 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:39.382004    5315 scope.go:122] "RemoveContainer" containerID="40d1300622a48f81b2e1a83b63f4b1be759c524180ae9c09713410b6db8814b1"
Nov 03 18:13:39.383760 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:39.383737    5315 generic.go:356] "Generic (PLEG): container finished" podID="36d4f867-6750-43de-8cdc-76f7e76fdd44" containerID="16228370c5964cfa3a4a65ddaca6c939b84eebd4df14ca5f5aa8eb65090781f3" exitCode=0
Nov 03 18:13:39.383869 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:39.383815    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r" event={"ID":"36d4f867-6750-43de-8cdc-76f7e76fdd44","Type":"ContainerDied","Data":"16228370c5964cfa3a4a65ddaca6c939b84eebd4df14ca5f5aa8eb65090781f3"}
Nov 03 18:13:39.384371 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:39.384342    5315 scope.go:122] "RemoveContainer" containerID="16228370c5964cfa3a4a65ddaca6c939b84eebd4df14ca5f5aa8eb65090781f3"
Nov 03 18:13:39.385896 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:39.385878    5315 generic.go:356] "Generic (PLEG): container finished" podID="a91acd67-7bae-4d10-a5b0-4f1d4d81253f" containerID="5a807df74fdb3586c786c96521db6b16a152470bd59c19a4be212fcd36135eaa" exitCode=0
Nov 03 18:13:39.386005 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:39.385958    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d" event={"ID":"a91acd67-7bae-4d10-a5b0-4f1d4d81253f","Type":"ContainerDied","Data":"5a807df74fdb3586c786c96521db6b16a152470bd59c19a4be212fcd36135eaa"}
Nov 03 18:13:39.386406 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:39.386366    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:13:39.386508 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:39.386380    5315 scope.go:122] "RemoveContainer" containerID="5a807df74fdb3586c786c96521db6b16a152470bd59c19a4be212fcd36135eaa"
Nov 03 18:13:39.386585 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:39.386470    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:13:39.386685 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:39.386641    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:13:39.387997 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:39.387969    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:13:39.388084 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:39.388049    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:13:39.388132 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:39.388097    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:13:39.388688 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:39.388667    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b6cb8403-77775ddc9f-pmzcv" event={"ID":"25bc2947-958b-4181-99a7-716108abe703","Type":"ContainerDied","Data":"37371535260f2f0afe7f16917151d9f8af253f854289d3a0cf99d5d21807d8f4"}
Nov 03 18:13:39.388746 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:39.388737    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b6cb8403-77775ddc9f-pmzcv"
Nov 03 18:13:39.394447 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:39.394424    5315 scope.go:122] "RemoveContainer" containerID="24ed2687b798f034b1d198ae91be48585c7d6e433657c5c342fa984a27fe7d02"
Nov 03 18:13:39.540082 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:39.540030    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b6cb8403-5545f5d6b7-6m5gp"]
Nov 03 18:13:39.545293 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:39.545255    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b6cb8403-5545f5d6b7-6m5gp"]
Nov 03 18:13:40.059105 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:40.059036    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b6cb8403-77775ddc9f-pmzcv"]
Nov 03 18:13:40.073499 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:40.073418    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b6cb8403-77775ddc9f-pmzcv"]
Nov 03 18:13:40.466684 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:40.428227    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r" event={"ID":"36d4f867-6750-43de-8cdc-76f7e76fdd44","Type":"ContainerStarted","Data":"8d83ae71521c7ad0906394baa523c3950a1510da5c32acae2d4dd04872610854"}
Nov 03 18:13:40.466684 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:40.437016    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d" event={"ID":"a91acd67-7bae-4d10-a5b0-4f1d4d81253f","Type":"ContainerStarted","Data":"82acbf6e071bfe45aedb1bd5e5941c1ef9889be7b7a1eba99f7849266106b367"}
Nov 03 18:13:40.554595 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:40.551040    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r"
Nov 03 18:13:40.579635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:40.567731    5315 patch_prober.go:29] interesting pod/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 03 18:13:40.579635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:40.567804    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r" podUID="36d4f867-6750-43de-8cdc-76f7e76fdd44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 03 18:13:40.609634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:40.607055    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d"
Nov 03 18:13:40.609634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:40.607187    5315 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Nov 03 18:13:40.609634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:40.607230    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d" podUID="a91acd67-7bae-4d10-a5b0-4f1d4d81253f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Nov 03 18:13:41.105736 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:41.105701    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25bc2947-958b-4181-99a7-716108abe703" path="/var/lib/kubelet/pods/25bc2947-958b-4181-99a7-716108abe703/volumes"
Nov 03 18:13:41.106188 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:41.106174    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="581c90bc-60cd-412a-b4d5-5bbab2b49336" path="/var/lib/kubelet/pods/581c90bc-60cd-412a-b4d5-5bbab2b49336/volumes"
Nov 03 18:13:41.546561 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:41.546092    5315 patch_prober.go:29] interesting pod/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 03 18:13:41.546561 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:41.546157    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r" podUID="36d4f867-6750-43de-8cdc-76f7e76fdd44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 03 18:13:41.605035 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:41.604993    5315 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Nov 03 18:13:41.605212 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:41.605051    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d" podUID="a91acd67-7bae-4d10-a5b0-4f1d4d81253f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Nov 03 18:13:42.563637 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:42.543172    5315 patch_prober.go:29] interesting pod/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 03 18:13:42.563637 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:42.543235    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r" podUID="36d4f867-6750-43de-8cdc-76f7e76fdd44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 03 18:13:42.605176 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:42.605133    5315 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Nov 03 18:13:42.605371 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:42.605205    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d" podUID="a91acd67-7bae-4d10-a5b0-4f1d4d81253f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Nov 03 18:13:43.092966 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:43.092897    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c91bee-76b997676c-zfjvq" podUID="1978e992-0367-40cf-a9f0-7899ab8215c9" containerName="payload-processing" containerID="cri-o://7297a1e5f173f8e7c0c8aa3aa585a358bd4e3c94132dc99dcb551298c51a49ba" gracePeriod=25
Nov 03 18:13:43.216557 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:43.216493    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c91bee-687764d75f-5dlqr" podUID="16103631-17f7-429b-b648-49691db9e8a5" containerName="payload-pre-processing" containerID="cri-o://2ca248c204797b3742e615508f42cced098bba4bbf8b2bc9393a702e545617dc" gracePeriod=25
Nov 03 18:13:43.493989 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:43.493955    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-0ccbb9-57dcc5ff7-zcv75"]
Nov 03 18:13:43.501407 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:43.501378    5315 generic.go:356] "Generic (PLEG): container finished" podID="16103631-17f7-429b-b648-49691db9e8a5" containerID="2ca248c204797b3742e615508f42cced098bba4bbf8b2bc9393a702e545617dc" exitCode=0
Nov 03 18:13:43.501646 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:43.501585    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c91bee-687764d75f-5dlqr" event={"ID":"16103631-17f7-429b-b648-49691db9e8a5","Type":"ContainerDied","Data":"2ca248c204797b3742e615508f42cced098bba4bbf8b2bc9393a702e545617dc"}
Nov 03 18:13:43.512188 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:43.510985    5315 generic.go:356] "Generic (PLEG): container finished" podID="1978e992-0367-40cf-a9f0-7899ab8215c9" containerID="7297a1e5f173f8e7c0c8aa3aa585a358bd4e3c94132dc99dcb551298c51a49ba" exitCode=0
Nov 03 18:13:43.512188 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:43.511068    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c91bee-76b997676c-zfjvq" event={"ID":"1978e992-0367-40cf-a9f0-7899ab8215c9","Type":"ContainerDied","Data":"7297a1e5f173f8e7c0c8aa3aa585a358bd4e3c94132dc99dcb551298c51a49ba"}
Nov 03 18:13:43.557633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:43.557043    5315 patch_prober.go:29] interesting pod/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 03 18:13:43.557633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:43.557112    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r" podUID="36d4f867-6750-43de-8cdc-76f7e76fdd44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 03 18:13:43.595699 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:43.595667    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c91bee-76b997676c-zfjvq"
Nov 03 18:13:43.607645 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:43.605694    5315 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Nov 03 18:13:43.607645 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:43.605762    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d" podUID="a91acd67-7bae-4d10-a5b0-4f1d4d81253f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Nov 03 18:13:43.622365 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:43.622324    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1978e992-0367-40cf-a9f0-7899ab8215c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1978e992-0367-40cf-a9f0-7899ab8215c9-plugins-config-volume\") pod \"1978e992-0367-40cf-a9f0-7899ab8215c9\" (UID: \"1978e992-0367-40cf-a9f0-7899ab8215c9\") "
Nov 03 18:13:43.622557 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:43.622402    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1978e992-0367-40cf-a9f0-7899ab8215c9-kube-api-access-5dgn5\" (UniqueName: \"kubernetes.io/projected/1978e992-0367-40cf-a9f0-7899ab8215c9-kube-api-access-5dgn5\") pod \"1978e992-0367-40cf-a9f0-7899ab8215c9\" (UID: \"1978e992-0367-40cf-a9f0-7899ab8215c9\") "
Nov 03 18:13:43.623155 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:43.623119    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1978e992-0367-40cf-a9f0-7899ab8215c9-plugins-config-volume" pod "1978e992-0367-40cf-a9f0-7899ab8215c9" (UID: "1978e992-0367-40cf-a9f0-7899ab8215c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:13:43.623267 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:43.623179    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-0ccbb9-6b4b8ccc7f-q47rc"]
Nov 03 18:13:43.695673 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:43.685366    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1978e992-0367-40cf-a9f0-7899ab8215c9-kube-api-access-5dgn5" pod "1978e992-0367-40cf-a9f0-7899ab8215c9" (UID: "1978e992-0367-40cf-a9f0-7899ab8215c9"). InnerVolumeSpecName "kube-api-access-5dgn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:13:43.709605 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:43.709576    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c91bee-687764d75f-5dlqr"
Nov 03 18:13:43.749683 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:43.749107    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16103631-17f7-429b-b648-49691db9e8a5-kube-api-access-vshlm\" (UniqueName: \"kubernetes.io/projected/16103631-17f7-429b-b648-49691db9e8a5-kube-api-access-vshlm\") pod \"16103631-17f7-429b-b648-49691db9e8a5\" (UID: \"16103631-17f7-429b-b648-49691db9e8a5\") "
Nov 03 18:13:43.749683 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:43.749176    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16103631-17f7-429b-b648-49691db9e8a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16103631-17f7-429b-b648-49691db9e8a5-plugins-config-volume\") pod \"16103631-17f7-429b-b648-49691db9e8a5\" (UID: \"16103631-17f7-429b-b648-49691db9e8a5\") "
Nov 03 18:13:43.749962 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:43.749906    5315 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1978e992-0367-40cf-a9f0-7899ab8215c9-plugins-config-volume\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:43.749962 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:43.749933    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dgn5\" (UniqueName: \"kubernetes.io/projected/1978e992-0367-40cf-a9f0-7899ab8215c9-kube-api-access-5dgn5\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:43.796646 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:43.795854    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16103631-17f7-429b-b648-49691db9e8a5-plugins-config-volume" pod "16103631-17f7-429b-b648-49691db9e8a5" (UID: "16103631-17f7-429b-b648-49691db9e8a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:13:43.803880 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:43.802727    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16103631-17f7-429b-b648-49691db9e8a5-kube-api-access-vshlm" pod "16103631-17f7-429b-b648-49691db9e8a5" (UID: "16103631-17f7-429b-b648-49691db9e8a5"). InnerVolumeSpecName "kube-api-access-vshlm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:13:43.852632 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:43.851862    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vshlm\" (UniqueName: \"kubernetes.io/projected/16103631-17f7-429b-b648-49691db9e8a5-kube-api-access-vshlm\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:43.852632 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:43.851904    5315 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16103631-17f7-429b-b648-49691db9e8a5-plugins-config-volume\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:44.521258 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:44.521163    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c91bee-687764d75f-5dlqr"
Nov 03 18:13:44.521258 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:44.521166    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c91bee-687764d75f-5dlqr" event={"ID":"16103631-17f7-429b-b648-49691db9e8a5","Type":"ContainerDied","Data":"627c76fba9566cee340847f1d33c3e1e06df86f1cc3a49c3a194e3ad8a2933f4"}
Nov 03 18:13:44.521489 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:44.521271    5315 scope.go:122] "RemoveContainer" containerID="2ca248c204797b3742e615508f42cced098bba4bbf8b2bc9393a702e545617dc"
Nov 03 18:13:44.523224 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:44.523028    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c91bee-76b997676c-zfjvq"
Nov 03 18:13:44.523224 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:44.523027    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c91bee-76b997676c-zfjvq" event={"ID":"1978e992-0367-40cf-a9f0-7899ab8215c9","Type":"ContainerDied","Data":"64bd3c2af8341fb1b5a2943fe207f0366f6618cdc561476ec687674845561141"}
Nov 03 18:13:44.541393 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:44.540877    5315 patch_prober.go:29] interesting pod/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 03 18:13:44.541393 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:44.540947    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r" podUID="36d4f867-6750-43de-8cdc-76f7e76fdd44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 03 18:13:44.584591 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:44.584566    5315 scope.go:122] "RemoveContainer" containerID="7297a1e5f173f8e7c0c8aa3aa585a358bd4e3c94132dc99dcb551298c51a49ba"
Nov 03 18:13:44.605443 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:44.605416    5315 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Nov 03 18:13:44.605805 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:44.605465    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d" podUID="a91acd67-7bae-4d10-a5b0-4f1d4d81253f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Nov 03 18:13:44.765231 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:44.765133    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-c91bee-76b997676c-zfjvq"]
Nov 03 18:13:44.782644 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:44.782151    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-c91bee-76b997676c-zfjvq"]
Nov 03 18:13:44.881408 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:44.880666    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c91bee-687764d75f-5dlqr"]
Nov 03 18:13:44.886666 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:44.886632    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c91bee-687764d75f-5dlqr"]
Nov 03 18:13:45.185737 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:45.182574    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16103631-17f7-429b-b648-49691db9e8a5" path="/var/lib/kubelet/pods/16103631-17f7-429b-b648-49691db9e8a5/volumes"
Nov 03 18:13:45.197632 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:45.193303    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1978e992-0367-40cf-a9f0-7899ab8215c9" path="/var/lib/kubelet/pods/1978e992-0367-40cf-a9f0-7899ab8215c9/volumes"
Nov 03 18:13:45.592626 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:45.588213    5315 patch_prober.go:29] interesting pod/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 03 18:13:45.592626 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:45.588278    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r" podUID="36d4f867-6750-43de-8cdc-76f7e76fdd44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 03 18:13:45.611632 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:45.607364    5315 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Nov 03 18:13:45.611632 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:45.607574    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d" podUID="a91acd67-7bae-4d10-a5b0-4f1d4d81253f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Nov 03 18:13:46.362835 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:46.362776    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d"
Nov 03 18:13:46.545635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:46.545070    5315 patch_prober.go:29] interesting pod/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 03 18:13:46.545635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:46.545151    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r" podUID="36d4f867-6750-43de-8cdc-76f7e76fdd44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 03 18:13:46.605412 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:46.605369    5315 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Nov 03 18:13:46.605578 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:46.605429    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d" podUID="a91acd67-7bae-4d10-a5b0-4f1d4d81253f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Nov 03 18:13:47.560032 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:47.559990    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r"
Nov 03 18:13:47.561637 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:47.561594    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r"
Nov 03 18:13:47.562802 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:47.562776    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r"
Nov 03 18:13:47.609790 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:47.609756    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d"
Nov 03 18:13:47.611034 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:47.611009    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d"
Nov 03 18:13:47.982474 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:47.982330    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b6cb8403-v7kr7"]
Nov 03 18:13:47.990830 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:47.990781    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b6cb8403-v7kr7"]
Nov 03 18:13:48.104876 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.104842    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56"]
Nov 03 18:13:48.105666 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.105602    5315 cpu_manager.go:414] "containerMap: removing container" podUID="25bc2947-958b-4181-99a7-716108abe703" containerName="payload-pre-processing"
Nov 03 18:13:48.105666 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.105673    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25bc2947-958b-4181-99a7-716108abe703" containerName="payload-pre-processing"
Nov 03 18:13:48.105878 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.105690    5315 cpu_manager.go:414] "containerMap: removing container" podUID="581c90bc-60cd-412a-b4d5-5bbab2b49336" containerName="payload-processing"
Nov 03 18:13:48.105878 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.105698    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="581c90bc-60cd-412a-b4d5-5bbab2b49336" containerName="payload-processing"
Nov 03 18:13:48.105878 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.105745    5315 cpu_manager.go:414] "containerMap: removing container" podUID="ad347a4e-f710-4482-a326-5e05d12b0dc3" containerName="revoke-keys"
Nov 03 18:13:48.105878 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.105752    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad347a4e-f710-4482-a326-5e05d12b0dc3" containerName="revoke-keys"
Nov 03 18:13:48.105878 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.105776    5315 cpu_manager.go:414] "containerMap: removing container" podUID="1978e992-0367-40cf-a9f0-7899ab8215c9" containerName="payload-processing"
Nov 03 18:13:48.105878 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.105784    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1978e992-0367-40cf-a9f0-7899ab8215c9" containerName="payload-processing"
Nov 03 18:13:48.105878 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.105800    5315 cpu_manager.go:414] "containerMap: removing container" podUID="16103631-17f7-429b-b648-49691db9e8a5" containerName="payload-pre-processing"
Nov 03 18:13:48.105878 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.105807    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16103631-17f7-429b-b648-49691db9e8a5" containerName="payload-pre-processing"
Nov 03 18:13:48.105878 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.105821    5315 cpu_manager.go:414] "containerMap: removing container" podUID="9569cd4c-0762-444e-88de-8e6825323878" containerName="revoke-keys"
Nov 03 18:13:48.105878 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.105827    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9569cd4c-0762-444e-88de-8e6825323878" containerName="revoke-keys"
Nov 03 18:13:48.106220 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.105985    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad347a4e-f710-4482-a326-5e05d12b0dc3" containerName="revoke-keys"
Nov 03 18:13:48.106220 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.105998    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="1978e992-0367-40cf-a9f0-7899ab8215c9" containerName="payload-processing"
Nov 03 18:13:48.106220 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.106005    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="16103631-17f7-429b-b648-49691db9e8a5" containerName="payload-pre-processing"
Nov 03 18:13:48.106220 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.106011    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="9569cd4c-0762-444e-88de-8e6825323878" containerName="revoke-keys"
Nov 03 18:13:48.106220 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.106022    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="25bc2947-958b-4181-99a7-716108abe703" containerName="payload-pre-processing"
Nov 03 18:13:48.106220 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.106029    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="581c90bc-60cd-412a-b4d5-5bbab2b49336" containerName="payload-processing"
Nov 03 18:13:48.109448 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.109420    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56"
Nov 03 18:13:48.130843 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.130811    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7296122e-d510-4826-9a6a-2f663d6603bf-workload-socket\") pod \"e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56\" (UID: \"7296122e-d510-4826-9a6a-2f663d6603bf\") " pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56"
Nov 03 18:13:48.130999 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.130918    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7296122e-d510-4826-9a6a-2f663d6603bf-istio-token\") pod \"e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56\" (UID: \"7296122e-d510-4826-9a6a-2f663d6603bf\") " pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56"
Nov 03 18:13:48.131055 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.131026    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7296122e-d510-4826-9a6a-2f663d6603bf-workload-certs\") pod \"e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56\" (UID: \"7296122e-d510-4826-9a6a-2f663d6603bf\") " pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56"
Nov 03 18:13:48.131108 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.131073    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7296122e-d510-4826-9a6a-2f663d6603bf-istio-podinfo\") pod \"e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56\" (UID: \"7296122e-d510-4826-9a6a-2f663d6603bf\") " pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56"
Nov 03 18:13:48.131242 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.131220    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7296122e-d510-4826-9a6a-2f663d6603bf-credential-socket\") pod \"e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56\" (UID: \"7296122e-d510-4826-9a6a-2f663d6603bf\") " pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56"
Nov 03 18:13:48.131297 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.131266    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7296122e-d510-4826-9a6a-2f663d6603bf-istio-envoy\") pod \"e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56\" (UID: \"7296122e-d510-4826-9a6a-2f663d6603bf\") " pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56"
Nov 03 18:13:48.131330 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.131298    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d6dk\" (UniqueName: \"kubernetes.io/projected/7296122e-d510-4826-9a6a-2f663d6603bf-kube-api-access-2d6dk\") pod \"e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56\" (UID: \"7296122e-d510-4826-9a6a-2f663d6603bf\") " pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56"
Nov 03 18:13:48.131389 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.131371    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7296122e-d510-4826-9a6a-2f663d6603bf-istio-data\") pod \"e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56\" (UID: \"7296122e-d510-4826-9a6a-2f663d6603bf\") " pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56"
Nov 03 18:13:48.131446 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.131425    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7296122e-d510-4826-9a6a-2f663d6603bf-istiod-ca-cert\") pod \"e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56\" (UID: \"7296122e-d510-4826-9a6a-2f663d6603bf\") " pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56"
Nov 03 18:13:48.161189 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.161065    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-c1b2d2-openshift-default-dockercfg-x68zb\""
Nov 03 18:13:48.178346 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.178303    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56"]
Nov 03 18:13:48.233226 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.233140    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d6dk\" (UniqueName: \"kubernetes.io/projected/7296122e-d510-4826-9a6a-2f663d6603bf-kube-api-access-2d6dk\") pod \"e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56\" (UID: \"7296122e-d510-4826-9a6a-2f663d6603bf\") " pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56"
Nov 03 18:13:48.233226 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.233192    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7296122e-d510-4826-9a6a-2f663d6603bf-istio-data\") pod \"e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56\" (UID: \"7296122e-d510-4826-9a6a-2f663d6603bf\") " pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56"
Nov 03 18:13:48.233226 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.233216    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7296122e-d510-4826-9a6a-2f663d6603bf-istiod-ca-cert\") pod \"e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56\" (UID: \"7296122e-d510-4826-9a6a-2f663d6603bf\") " pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56"
Nov 03 18:13:48.233533 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.233263    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7296122e-d510-4826-9a6a-2f663d6603bf-workload-socket\") pod \"e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56\" (UID: \"7296122e-d510-4826-9a6a-2f663d6603bf\") " pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56"
Nov 03 18:13:48.233533 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.233399    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7296122e-d510-4826-9a6a-2f663d6603bf-istio-token\") pod \"e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56\" (UID: \"7296122e-d510-4826-9a6a-2f663d6603bf\") " pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56"
Nov 03 18:13:48.233533 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.233522    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7296122e-d510-4826-9a6a-2f663d6603bf-workload-certs\") pod \"e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56\" (UID: \"7296122e-d510-4826-9a6a-2f663d6603bf\") " pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56"
Nov 03 18:13:48.233750 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.233676    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7296122e-d510-4826-9a6a-2f663d6603bf-istio-podinfo\") pod \"e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56\" (UID: \"7296122e-d510-4826-9a6a-2f663d6603bf\") " pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56"
Nov 03 18:13:48.233804 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.233737    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7296122e-d510-4826-9a6a-2f663d6603bf-istio-data\") pod \"e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56\" (UID: \"7296122e-d510-4826-9a6a-2f663d6603bf\") " pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56"
Nov 03 18:13:48.233804 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.233755    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7296122e-d510-4826-9a6a-2f663d6603bf-workload-socket\") pod \"e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56\" (UID: \"7296122e-d510-4826-9a6a-2f663d6603bf\") " pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56"
Nov 03 18:13:48.233881 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.233835    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7296122e-d510-4826-9a6a-2f663d6603bf-credential-socket\") pod \"e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56\" (UID: \"7296122e-d510-4826-9a6a-2f663d6603bf\") " pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56"
Nov 03 18:13:48.233881 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.233863    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7296122e-d510-4826-9a6a-2f663d6603bf-istio-envoy\") pod \"e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56\" (UID: \"7296122e-d510-4826-9a6a-2f663d6603bf\") " pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56"
Nov 03 18:13:48.233974 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.233943    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7296122e-d510-4826-9a6a-2f663d6603bf-workload-certs\") pod \"e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56\" (UID: \"7296122e-d510-4826-9a6a-2f663d6603bf\") " pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56"
Nov 03 18:13:48.234156 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.234135    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7296122e-d510-4826-9a6a-2f663d6603bf-istiod-ca-cert\") pod \"e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56\" (UID: \"7296122e-d510-4826-9a6a-2f663d6603bf\") " pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56"
Nov 03 18:13:48.234217 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.234167    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7296122e-d510-4826-9a6a-2f663d6603bf-credential-socket\") pod \"e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56\" (UID: \"7296122e-d510-4826-9a6a-2f663d6603bf\") " pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56"
Nov 03 18:13:48.235992 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.235966    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7296122e-d510-4826-9a6a-2f663d6603bf-istio-envoy\") pod \"e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56\" (UID: \"7296122e-d510-4826-9a6a-2f663d6603bf\") " pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56"
Nov 03 18:13:48.236121 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.236098    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7296122e-d510-4826-9a6a-2f663d6603bf-istio-podinfo\") pod \"e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56\" (UID: \"7296122e-d510-4826-9a6a-2f663d6603bf\") " pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56"
Nov 03 18:13:48.320962 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.320307    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7296122e-d510-4826-9a6a-2f663d6603bf-istio-token\") pod \"e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56\" (UID: \"7296122e-d510-4826-9a6a-2f663d6603bf\") " pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56"
Nov 03 18:13:48.326643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.323051    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d6dk\" (UniqueName: \"kubernetes.io/projected/7296122e-d510-4826-9a6a-2f663d6603bf-kube-api-access-2d6dk\") pod \"e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56\" (UID: \"7296122e-d510-4826-9a6a-2f663d6603bf\") " pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56"
Nov 03 18:13:48.426634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.423007    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56"
Nov 03 18:13:48.498466 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.497707    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-0ccbb9-57dcc5ff7-zcv75" podUID="b86eb9eb-969a-4ac5-a947-0f1d32ca5b72" containerName="payload-processing" containerID="cri-o://739bb1e5c2fba8967292d6a0db43d1eda8dfcd7af96cf7a2719ff6ca72202205" gracePeriod=25
Nov 03 18:13:48.624103 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.624059    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0ccbb9-6b4b8ccc7f-q47rc" podUID="e7580e5e-cc8b-412e-99f0-3a71c762e618" containerName="payload-pre-processing" containerID="cri-o://4f586d9a86d82eca3d2d65bae3fae862f698adcdf5d24d6628fd2d81bbc45a69" gracePeriod=25
Nov 03 18:13:48.877773 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.877733    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s" podUID="155e4b39-1877-4bd9-8541-5d6bd3aa0a8e" containerName="main" containerID="cri-o://b45322fa2f2e8a299cd8514f46f06c912e78288d420ef1555b58f0746a8f5813" gracePeriod=45
Nov 03 18:13:48.903299 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.903246    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56"]
Nov 03 18:13:48.926705 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:13:48.926665    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7296122e_d510_4826_9a6a_2f663d6603bf.slice/crio-7417c0bea5ea62ca6863e80e36b2c4f0dbb923faad6b84931debeaf4a16d4eb2 WatchSource:0}: Error finding container 7417c0bea5ea62ca6863e80e36b2c4f0dbb923faad6b84931debeaf4a16d4eb2: Status 404 returned error can't find the container with id 7417c0bea5ea62ca6863e80e36b2c4f0dbb923faad6b84931debeaf4a16d4eb2
Nov 03 18:13:48.983579 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.983539    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:13:48.983710 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.983631    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:13:48.983710 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:48.983665    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:13:49.109634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:49.109276    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45a6b9f7-0d90-448f-a209-5f5517530e72" path="/var/lib/kubelet/pods/45a6b9f7-0d90-448f-a209-5f5517530e72/volumes"
Nov 03 18:13:49.178847 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:49.178816    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s"
Nov 03 18:13:49.265277 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:49.265248    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/155e4b39-1877-4bd9-8541-5d6bd3aa0a8e-home\" (UniqueName: \"kubernetes.io/empty-dir/155e4b39-1877-4bd9-8541-5d6bd3aa0a8e-home\") pod \"155e4b39-1877-4bd9-8541-5d6bd3aa0a8e\" (UID: \"155e4b39-1877-4bd9-8541-5d6bd3aa0a8e\") "
Nov 03 18:13:49.265421 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:49.265301    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/155e4b39-1877-4bd9-8541-5d6bd3aa0a8e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/155e4b39-1877-4bd9-8541-5d6bd3aa0a8e-dshm\") pod \"155e4b39-1877-4bd9-8541-5d6bd3aa0a8e\" (UID: \"155e4b39-1877-4bd9-8541-5d6bd3aa0a8e\") "
Nov 03 18:13:49.265421 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:49.265375    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/155e4b39-1877-4bd9-8541-5d6bd3aa0a8e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/155e4b39-1877-4bd9-8541-5d6bd3aa0a8e-tmp-dir\") pod \"155e4b39-1877-4bd9-8541-5d6bd3aa0a8e\" (UID: \"155e4b39-1877-4bd9-8541-5d6bd3aa0a8e\") "
Nov 03 18:13:49.265564 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:49.265501    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/155e4b39-1877-4bd9-8541-5d6bd3aa0a8e-kube-api-access-7fnhz\" (UniqueName: \"kubernetes.io/projected/155e4b39-1877-4bd9-8541-5d6bd3aa0a8e-kube-api-access-7fnhz\") pod \"155e4b39-1877-4bd9-8541-5d6bd3aa0a8e\" (UID: \"155e4b39-1877-4bd9-8541-5d6bd3aa0a8e\") "
Nov 03 18:13:49.265564 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:49.265522    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/155e4b39-1877-4bd9-8541-5d6bd3aa0a8e-home" pod "155e4b39-1877-4bd9-8541-5d6bd3aa0a8e" (UID: "155e4b39-1877-4bd9-8541-5d6bd3aa0a8e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:13:49.265695 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:49.265661    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/155e4b39-1877-4bd9-8541-5d6bd3aa0a8e-tmp-dir" pod "155e4b39-1877-4bd9-8541-5d6bd3aa0a8e" (UID: "155e4b39-1877-4bd9-8541-5d6bd3aa0a8e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:13:49.265695 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:49.265675    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/155e4b39-1877-4bd9-8541-5d6bd3aa0a8e-tls-certs\" (UniqueName: \"kubernetes.io/secret/155e4b39-1877-4bd9-8541-5d6bd3aa0a8e-tls-certs\") pod \"155e4b39-1877-4bd9-8541-5d6bd3aa0a8e\" (UID: \"155e4b39-1877-4bd9-8541-5d6bd3aa0a8e\") "
Nov 03 18:13:49.265816 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:49.265795    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/155e4b39-1877-4bd9-8541-5d6bd3aa0a8e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/155e4b39-1877-4bd9-8541-5d6bd3aa0a8e-model-cache\") pod \"155e4b39-1877-4bd9-8541-5d6bd3aa0a8e\" (UID: \"155e4b39-1877-4bd9-8541-5d6bd3aa0a8e\") "
Nov 03 18:13:49.266081 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:49.266059    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/155e4b39-1877-4bd9-8541-5d6bd3aa0a8e-model-cache" pod "155e4b39-1877-4bd9-8541-5d6bd3aa0a8e" (UID: "155e4b39-1877-4bd9-8541-5d6bd3aa0a8e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:13:49.266416 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:49.266398    5315 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/155e4b39-1877-4bd9-8541-5d6bd3aa0a8e-home\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:49.266480 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:49.266422    5315 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/155e4b39-1877-4bd9-8541-5d6bd3aa0a8e-tmp-dir\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:49.266480 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:49.266439    5315 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/155e4b39-1877-4bd9-8541-5d6bd3aa0a8e-model-cache\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:49.267676 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:49.267649    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/155e4b39-1877-4bd9-8541-5d6bd3aa0a8e-dshm" pod "155e4b39-1877-4bd9-8541-5d6bd3aa0a8e" (UID: "155e4b39-1877-4bd9-8541-5d6bd3aa0a8e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:13:49.267772 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:49.267743    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/155e4b39-1877-4bd9-8541-5d6bd3aa0a8e-tls-certs" pod "155e4b39-1877-4bd9-8541-5d6bd3aa0a8e" (UID: "155e4b39-1877-4bd9-8541-5d6bd3aa0a8e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:13:49.268007 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:49.267988    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/155e4b39-1877-4bd9-8541-5d6bd3aa0a8e-kube-api-access-7fnhz" pod "155e4b39-1877-4bd9-8541-5d6bd3aa0a8e" (UID: "155e4b39-1877-4bd9-8541-5d6bd3aa0a8e"). InnerVolumeSpecName "kube-api-access-7fnhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:13:49.368380 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:49.368338    5315 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/155e4b39-1877-4bd9-8541-5d6bd3aa0a8e-dshm\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:49.368380 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:49.368384    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fnhz\" (UniqueName: \"kubernetes.io/projected/155e4b39-1877-4bd9-8541-5d6bd3aa0a8e-kube-api-access-7fnhz\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:49.368587 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:49.368401    5315 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/155e4b39-1877-4bd9-8541-5d6bd3aa0a8e-tls-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:49.644582 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:49.644541    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56" event={"ID":"7296122e-d510-4826-9a6a-2f663d6603bf","Type":"ContainerStarted","Data":"4defaaf0656c36b5a0a57ca43a2b83dd12eaff89e5a3fe5d8ca68d955cf57dd1"}
Nov 03 18:13:49.645147 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:49.644623    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56" event={"ID":"7296122e-d510-4826-9a6a-2f663d6603bf","Type":"ContainerStarted","Data":"7417c0bea5ea62ca6863e80e36b2c4f0dbb923faad6b84931debeaf4a16d4eb2"}
Nov 03 18:13:49.646192 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:49.646163    5315 generic.go:356] "Generic (PLEG): container finished" podID="155e4b39-1877-4bd9-8541-5d6bd3aa0a8e" containerID="b45322fa2f2e8a299cd8514f46f06c912e78288d420ef1555b58f0746a8f5813" exitCode=0
Nov 03 18:13:49.646300 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:49.646229    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s" event={"ID":"155e4b39-1877-4bd9-8541-5d6bd3aa0a8e","Type":"ContainerDied","Data":"b45322fa2f2e8a299cd8514f46f06c912e78288d420ef1555b58f0746a8f5813"}
Nov 03 18:13:49.646300 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:49.646252    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s" event={"ID":"155e4b39-1877-4bd9-8541-5d6bd3aa0a8e","Type":"ContainerDied","Data":"e7563dba898af3db10bb674438756cd7fbaa27b4ac7df591b3b48fa3dffa0526"}
Nov 03 18:13:49.646300 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:49.646269    5315 scope.go:122] "RemoveContainer" containerID="b45322fa2f2e8a299cd8514f46f06c912e78288d420ef1555b58f0746a8f5813"
Nov 03 18:13:49.646436 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:49.646421    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s"
Nov 03 18:13:49.657128 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:49.657109    5315 scope.go:122] "RemoveContainer" containerID="b45322fa2f2e8a299cd8514f46f06c912e78288d420ef1555b58f0746a8f5813"
Nov 03 18:13:49.657414 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:13:49.657393    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b45322fa2f2e8a299cd8514f46f06c912e78288d420ef1555b58f0746a8f5813\": container with ID starting with b45322fa2f2e8a299cd8514f46f06c912e78288d420ef1555b58f0746a8f5813 not found: ID does not exist" containerID="b45322fa2f2e8a299cd8514f46f06c912e78288d420ef1555b58f0746a8f5813"
Nov 03 18:13:49.657455 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:49.657421    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b45322fa2f2e8a299cd8514f46f06c912e78288d420ef1555b58f0746a8f5813"} err="failed to get container status \"b45322fa2f2e8a299cd8514f46f06c912e78288d420ef1555b58f0746a8f5813\": rpc error: code = NotFound desc = could not find container \"b45322fa2f2e8a299cd8514f46f06c912e78288d420ef1555b58f0746a8f5813\": container with ID starting with b45322fa2f2e8a299cd8514f46f06c912e78288d420ef1555b58f0746a8f5813 not found: ID does not exist"
Nov 03 18:13:49.762685 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:49.762605    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56" podStartSLOduration=1.762590596 podStartE2EDuration="1.762590596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:13:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:13:49.759502617 +0000 UTC m=+1668.775700687" watchObservedRunningTime="2026-11-03 18:13:49.762590596 +0000 UTC m=+1668.778789012"
Nov 03 18:13:49.865092 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:49.865051    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s"]
Nov 03 18:13:49.869400 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:49.869367    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-0ccbb9/distinct-2-w0-0ccbb9-kserve-8645fcb8cf-9pw8s"]
Nov 03 18:13:50.423368 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:50.423335    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56"
Nov 03 18:13:50.429348 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:50.429319    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56"
Nov 03 18:13:50.665439 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:50.665407    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56"
Nov 03 18:13:50.666439 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:50.666414    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56"
Nov 03 18:13:51.108388 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:51.108353    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="155e4b39-1877-4bd9-8541-5d6bd3aa0a8e" path="/var/lib/kubelet/pods/155e4b39-1877-4bd9-8541-5d6bd3aa0a8e/volumes"
Nov 03 18:13:51.468349 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:51.468264    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f52dedf-zlhqr"]
Nov 03 18:13:51.468998 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:51.468971    5315 cpu_manager.go:414] "containerMap: removing container" podUID="155e4b39-1877-4bd9-8541-5d6bd3aa0a8e" containerName="main"
Nov 03 18:13:51.468998 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:51.468992    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="155e4b39-1877-4bd9-8541-5d6bd3aa0a8e" containerName="main"
Nov 03 18:13:51.469200 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:51.469185    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="155e4b39-1877-4bd9-8541-5d6bd3aa0a8e" containerName="main"
Nov 03 18:13:51.472674 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:51.472653    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f52dedf-zlhqr"
Nov 03 18:13:51.520430 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:51.520396    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfefd039-svlhc"]
Nov 03 18:13:51.524409 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:51.524383    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfefd039-svlhc"
Nov 03 18:13:51.563200 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:51.563163    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:13:51.563200 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:51.563198    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:13:51.567297 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:51.567267    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:13:51.567297 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:51.567276    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:13:51.570934 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:51.570908    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f52dedf-zlhqr"]
Nov 03 18:13:51.614154 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:51.613747    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afa2c09e-863f-4086-b3bb-990dc6fe9b94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfefd039-svlhc\" (UID: \"afa2c09e-863f-4086-b3bb-990dc6fe9b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfefd039-svlhc"
Nov 03 18:13:51.615718 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:51.615679    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d467951-e1e8-4900-8d5f-ba26ee79bdd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f52dedf-zlhqr\" (UID: \"6d467951-e1e8-4900-8d5f-ba26ee79bdd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f52dedf-zlhqr"
Nov 03 18:13:51.615873 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:51.615766    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2nn8\" (UniqueName: \"kubernetes.io/projected/afa2c09e-863f-4086-b3bb-990dc6fe9b94-kube-api-access-t2nn8\") pod \"maas-api-revoke-sub-dfefd039-svlhc\" (UID: \"afa2c09e-863f-4086-b3bb-990dc6fe9b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfefd039-svlhc"
Nov 03 18:13:51.615944 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:51.615919    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afa2c09e-863f-4086-b3bb-990dc6fe9b94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfefd039-svlhc\" (UID: \"afa2c09e-863f-4086-b3bb-990dc6fe9b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfefd039-svlhc"
Nov 03 18:13:51.616013 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:51.615988    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d467951-e1e8-4900-8d5f-ba26ee79bdd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f52dedf-zlhqr\" (UID: \"6d467951-e1e8-4900-8d5f-ba26ee79bdd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f52dedf-zlhqr"
Nov 03 18:13:51.616095 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:51.616079    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lwc6\" (UniqueName: \"kubernetes.io/projected/6d467951-e1e8-4900-8d5f-ba26ee79bdd6-kube-api-access-6lwc6\") pod \"maas-api-revoke-sub-2f52dedf-zlhqr\" (UID: \"6d467951-e1e8-4900-8d5f-ba26ee79bdd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f52dedf-zlhqr"
Nov 03 18:13:51.621456 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:51.621423    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfefd039-svlhc"]
Nov 03 18:13:51.717365 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:51.717328    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d467951-e1e8-4900-8d5f-ba26ee79bdd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f52dedf-zlhqr\" (UID: \"6d467951-e1e8-4900-8d5f-ba26ee79bdd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f52dedf-zlhqr"
Nov 03 18:13:51.717867 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:51.717406    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2nn8\" (UniqueName: \"kubernetes.io/projected/afa2c09e-863f-4086-b3bb-990dc6fe9b94-kube-api-access-t2nn8\") pod \"maas-api-revoke-sub-dfefd039-svlhc\" (UID: \"afa2c09e-863f-4086-b3bb-990dc6fe9b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfefd039-svlhc"
Nov 03 18:13:51.717867 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:51.717522    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afa2c09e-863f-4086-b3bb-990dc6fe9b94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfefd039-svlhc\" (UID: \"afa2c09e-863f-4086-b3bb-990dc6fe9b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfefd039-svlhc"
Nov 03 18:13:51.717867 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:51.717562    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d467951-e1e8-4900-8d5f-ba26ee79bdd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f52dedf-zlhqr\" (UID: \"6d467951-e1e8-4900-8d5f-ba26ee79bdd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f52dedf-zlhqr"
Nov 03 18:13:51.717867 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:51.717645    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lwc6\" (UniqueName: \"kubernetes.io/projected/6d467951-e1e8-4900-8d5f-ba26ee79bdd6-kube-api-access-6lwc6\") pod \"maas-api-revoke-sub-2f52dedf-zlhqr\" (UID: \"6d467951-e1e8-4900-8d5f-ba26ee79bdd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f52dedf-zlhqr"
Nov 03 18:13:51.717867 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:51.717728    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afa2c09e-863f-4086-b3bb-990dc6fe9b94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfefd039-svlhc\" (UID: \"afa2c09e-863f-4086-b3bb-990dc6fe9b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfefd039-svlhc"
Nov 03 18:13:51.720700 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:51.720552    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d467951-e1e8-4900-8d5f-ba26ee79bdd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f52dedf-zlhqr\" (UID: \"6d467951-e1e8-4900-8d5f-ba26ee79bdd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f52dedf-zlhqr"
Nov 03 18:13:51.720974 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:51.720951    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afa2c09e-863f-4086-b3bb-990dc6fe9b94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfefd039-svlhc\" (UID: \"afa2c09e-863f-4086-b3bb-990dc6fe9b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfefd039-svlhc"
Nov 03 18:13:51.721238 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:51.721215    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d467951-e1e8-4900-8d5f-ba26ee79bdd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f52dedf-zlhqr\" (UID: \"6d467951-e1e8-4900-8d5f-ba26ee79bdd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f52dedf-zlhqr"
Nov 03 18:13:51.721733 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:51.721711    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afa2c09e-863f-4086-b3bb-990dc6fe9b94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfefd039-svlhc\" (UID: \"afa2c09e-863f-4086-b3bb-990dc6fe9b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfefd039-svlhc"
Nov 03 18:13:51.775126 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:51.775092    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2nn8\" (UniqueName: \"kubernetes.io/projected/afa2c09e-863f-4086-b3bb-990dc6fe9b94-kube-api-access-t2nn8\") pod \"maas-api-revoke-sub-dfefd039-svlhc\" (UID: \"afa2c09e-863f-4086-b3bb-990dc6fe9b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfefd039-svlhc"
Nov 03 18:13:51.775285 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:51.775183    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lwc6\" (UniqueName: \"kubernetes.io/projected/6d467951-e1e8-4900-8d5f-ba26ee79bdd6-kube-api-access-6lwc6\") pod \"maas-api-revoke-sub-2f52dedf-zlhqr\" (UID: \"6d467951-e1e8-4900-8d5f-ba26ee79bdd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f52dedf-zlhqr"
Nov 03 18:13:51.806681 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:51.789976    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f52dedf-zlhqr"
Nov 03 18:13:51.857192 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:51.853353    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfefd039-svlhc"
Nov 03 18:13:52.372109 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:52.372071    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfefd039-svlhc"]
Nov 03 18:13:52.377558 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:13:52.377522    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafa2c09e_863f_4086_b3bb_990dc6fe9b94.slice/crio-74d05f56771cfda4f95bafeabc1da73a467787438ecf58c33a7594861eadca8f WatchSource:0}: Error finding container 74d05f56771cfda4f95bafeabc1da73a467787438ecf58c33a7594861eadca8f: Status 404 returned error can't find the container with id 74d05f56771cfda4f95bafeabc1da73a467787438ecf58c33a7594861eadca8f
Nov 03 18:13:52.422933 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:52.422903    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f52dedf-zlhqr"]
Nov 03 18:13:52.428145 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:13:52.428117    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d467951_e1e8_4900_8d5f_ba26ee79bdd6.slice/crio-5c17e6a3ceb7c9cda370e2ddd7d2a01b4f14546f3266986fcd762c623722c21d WatchSource:0}: Error finding container 5c17e6a3ceb7c9cda370e2ddd7d2a01b4f14546f3266986fcd762c623722c21d: Status 404 returned error can't find the container with id 5c17e6a3ceb7c9cda370e2ddd7d2a01b4f14546f3266986fcd762c623722c21d
Nov 03 18:13:52.745645 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:52.745532    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfefd039-svlhc" event={"ID":"afa2c09e-863f-4086-b3bb-990dc6fe9b94","Type":"ContainerStarted","Data":"74d05f56771cfda4f95bafeabc1da73a467787438ecf58c33a7594861eadca8f"}
Nov 03 18:13:52.793028 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:52.791066    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f52dedf-zlhqr" event={"ID":"6d467951-e1e8-4900-8d5f-ba26ee79bdd6","Type":"ContainerStarted","Data":"5c17e6a3ceb7c9cda370e2ddd7d2a01b4f14546f3266986fcd762c623722c21d"}
Nov 03 18:13:53.817964 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:53.817921    5315 generic.go:356] "Generic (PLEG): container finished" podID="afa2c09e-863f-4086-b3bb-990dc6fe9b94" containerID="c4ba43618a3b724d6ee5861a3cdb4ac8c748f9a2f19b4ef34518c01273caa558" exitCode=0
Nov 03 18:13:53.818393 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:53.817999    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfefd039-svlhc" event={"ID":"afa2c09e-863f-4086-b3bb-990dc6fe9b94","Type":"ContainerDied","Data":"c4ba43618a3b724d6ee5861a3cdb4ac8c748f9a2f19b4ef34518c01273caa558"}
Nov 03 18:13:53.819721 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:53.819688    5315 generic.go:356] "Generic (PLEG): container finished" podID="6d467951-e1e8-4900-8d5f-ba26ee79bdd6" containerID="03fca753ee86d38e5a9d763b20ba7e1e9240404937517e3ce0d54fc53c78b1f0" exitCode=0
Nov 03 18:13:53.819845 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:53.819774    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f52dedf-zlhqr" event={"ID":"6d467951-e1e8-4900-8d5f-ba26ee79bdd6","Type":"ContainerDied","Data":"03fca753ee86d38e5a9d763b20ba7e1e9240404937517e3ce0d54fc53c78b1f0"}
Nov 03 18:13:53.869016 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:53.868981    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c91bee-5vfdm"]
Nov 03 18:13:53.915442 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:53.915413    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c91bee-5vfdm"]
Nov 03 18:13:54.509643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:54.509578    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0ccbb9-6b4b8ccc7f-q47rc" podUID="e7580e5e-cc8b-412e-99f0-3a71c762e618" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:13:54.509829 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:54.509587    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-0ccbb9-57dcc5ff7-zcv75" podUID="b86eb9eb-969a-4ac5-a947-0f1d32ca5b72" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:13:54.578949 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:13:54.578914    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:13:55.023274 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:55.023247    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfefd039-svlhc"
Nov 03 18:13:55.026970 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:55.026945    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f52dedf-zlhqr"
Nov 03 18:13:55.106334 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:55.106253    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9569cd4c-0762-444e-88de-8e6825323878" path="/var/lib/kubelet/pods/9569cd4c-0762-444e-88de-8e6825323878/volumes"
Nov 03 18:13:55.118412 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:55.118383    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afa2c09e-863f-4086-b3bb-990dc6fe9b94-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afa2c09e-863f-4086-b3bb-990dc6fe9b94-maas-api-openshift-ca\") pod \"afa2c09e-863f-4086-b3bb-990dc6fe9b94\" (UID: \"afa2c09e-863f-4086-b3bb-990dc6fe9b94\") "
Nov 03 18:13:55.118645 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:55.118433    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afa2c09e-863f-4086-b3bb-990dc6fe9b94-kube-api-access-t2nn8\" (UniqueName: \"kubernetes.io/projected/afa2c09e-863f-4086-b3bb-990dc6fe9b94-kube-api-access-t2nn8\") pod \"afa2c09e-863f-4086-b3bb-990dc6fe9b94\" (UID: \"afa2c09e-863f-4086-b3bb-990dc6fe9b94\") "
Nov 03 18:13:55.118645 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:55.118462    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d467951-e1e8-4900-8d5f-ba26ee79bdd6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d467951-e1e8-4900-8d5f-ba26ee79bdd6-maas-api-openshift-ca\") pod \"6d467951-e1e8-4900-8d5f-ba26ee79bdd6\" (UID: \"6d467951-e1e8-4900-8d5f-ba26ee79bdd6\") "
Nov 03 18:13:55.118645 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:55.118501    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afa2c09e-863f-4086-b3bb-990dc6fe9b94-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afa2c09e-863f-4086-b3bb-990dc6fe9b94-maas-api-xks-ca\") pod \"afa2c09e-863f-4086-b3bb-990dc6fe9b94\" (UID: \"afa2c09e-863f-4086-b3bb-990dc6fe9b94\") "
Nov 03 18:13:55.118645 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:55.118569    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d467951-e1e8-4900-8d5f-ba26ee79bdd6-kube-api-access-6lwc6\" (UniqueName: \"kubernetes.io/projected/6d467951-e1e8-4900-8d5f-ba26ee79bdd6-kube-api-access-6lwc6\") pod \"6d467951-e1e8-4900-8d5f-ba26ee79bdd6\" (UID: \"6d467951-e1e8-4900-8d5f-ba26ee79bdd6\") "
Nov 03 18:13:55.118645 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:55.118604    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d467951-e1e8-4900-8d5f-ba26ee79bdd6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d467951-e1e8-4900-8d5f-ba26ee79bdd6-maas-api-xks-ca\") pod \"6d467951-e1e8-4900-8d5f-ba26ee79bdd6\" (UID: \"6d467951-e1e8-4900-8d5f-ba26ee79bdd6\") "
Nov 03 18:13:55.118965 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:55.118774    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afa2c09e-863f-4086-b3bb-990dc6fe9b94-maas-api-openshift-ca" pod "afa2c09e-863f-4086-b3bb-990dc6fe9b94" (UID: "afa2c09e-863f-4086-b3bb-990dc6fe9b94"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:13:55.118965 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:55.118869    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d467951-e1e8-4900-8d5f-ba26ee79bdd6-maas-api-openshift-ca" pod "6d467951-e1e8-4900-8d5f-ba26ee79bdd6" (UID: "6d467951-e1e8-4900-8d5f-ba26ee79bdd6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:13:55.119322 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:55.119298    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afa2c09e-863f-4086-b3bb-990dc6fe9b94-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:55.119447 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:55.119326    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d467951-e1e8-4900-8d5f-ba26ee79bdd6-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:55.120827 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:55.120805    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d467951-e1e8-4900-8d5f-ba26ee79bdd6-maas-api-xks-ca" pod "6d467951-e1e8-4900-8d5f-ba26ee79bdd6" (UID: "6d467951-e1e8-4900-8d5f-ba26ee79bdd6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:13:55.120827 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:55.120813    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d467951-e1e8-4900-8d5f-ba26ee79bdd6-kube-api-access-6lwc6" pod "6d467951-e1e8-4900-8d5f-ba26ee79bdd6" (UID: "6d467951-e1e8-4900-8d5f-ba26ee79bdd6"). InnerVolumeSpecName "kube-api-access-6lwc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:13:55.121213 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:55.121189    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afa2c09e-863f-4086-b3bb-990dc6fe9b94-maas-api-xks-ca" pod "afa2c09e-863f-4086-b3bb-990dc6fe9b94" (UID: "afa2c09e-863f-4086-b3bb-990dc6fe9b94"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:13:55.121281 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:55.121241    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afa2c09e-863f-4086-b3bb-990dc6fe9b94-kube-api-access-t2nn8" pod "afa2c09e-863f-4086-b3bb-990dc6fe9b94" (UID: "afa2c09e-863f-4086-b3bb-990dc6fe9b94"). InnerVolumeSpecName "kube-api-access-t2nn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:13:55.222632 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:55.222407    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lwc6\" (UniqueName: \"kubernetes.io/projected/6d467951-e1e8-4900-8d5f-ba26ee79bdd6-kube-api-access-6lwc6\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:55.222632 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:55.222459    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d467951-e1e8-4900-8d5f-ba26ee79bdd6-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:55.222632 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:55.222482    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2nn8\" (UniqueName: \"kubernetes.io/projected/afa2c09e-863f-4086-b3bb-990dc6fe9b94-kube-api-access-t2nn8\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:55.222632 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:55.222504    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afa2c09e-863f-4086-b3bb-990dc6fe9b94-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:55.468462 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:55.468379    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w"]
Nov 03 18:13:55.468689 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:55.468658    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w" podUID="f85c57a2-8cb8-4403-ba55-1de5bf63a921" containerName="istio-proxy" containerID="cri-o://174a67c0fd81c58870ea0e9570ac58ddb00de44b552e94daab9ad7c2c61cf31a" gracePeriod=30
Nov 03 18:13:55.877811 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:55.876288    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfefd039-svlhc"
Nov 03 18:13:55.877811 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:55.876287    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfefd039-svlhc" event={"ID":"afa2c09e-863f-4086-b3bb-990dc6fe9b94","Type":"ContainerDied","Data":"74d05f56771cfda4f95bafeabc1da73a467787438ecf58c33a7594861eadca8f"}
Nov 03 18:13:55.877811 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:55.876424    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74d05f56771cfda4f95bafeabc1da73a467787438ecf58c33a7594861eadca8f"
Nov 03 18:13:55.878147 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:55.878126    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f52dedf-zlhqr" event={"ID":"6d467951-e1e8-4900-8d5f-ba26ee79bdd6","Type":"ContainerDied","Data":"5c17e6a3ceb7c9cda370e2ddd7d2a01b4f14546f3266986fcd762c623722c21d"}
Nov 03 18:13:55.878212 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:55.878174    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c17e6a3ceb7c9cda370e2ddd7d2a01b4f14546f3266986fcd762c623722c21d"
Nov 03 18:13:55.878212 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:55.878179    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f52dedf-zlhqr"
Nov 03 18:13:56.330519 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:56.330483    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c1b2d2-6557dc56cf-kc2p2"]
Nov 03 18:13:56.339096 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:56.336378    5315 cpu_manager.go:414] "containerMap: removing container" podUID="6d467951-e1e8-4900-8d5f-ba26ee79bdd6" containerName="revoke-keys"
Nov 03 18:13:56.339339 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:56.339314    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d467951-e1e8-4900-8d5f-ba26ee79bdd6" containerName="revoke-keys"
Nov 03 18:13:56.339651 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:56.339628    5315 cpu_manager.go:414] "containerMap: removing container" podUID="afa2c09e-863f-4086-b3bb-990dc6fe9b94" containerName="revoke-keys"
Nov 03 18:13:56.339773 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:56.339759    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afa2c09e-863f-4086-b3bb-990dc6fe9b94" containerName="revoke-keys"
Nov 03 18:13:56.340111 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:56.340095    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d467951-e1e8-4900-8d5f-ba26ee79bdd6" containerName="revoke-keys"
Nov 03 18:13:56.358687 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:56.345970    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="afa2c09e-863f-4086-b3bb-990dc6fe9b94" containerName="revoke-keys"
Nov 03 18:13:56.368249 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:56.368211    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c1b2d2-6557dc56cf-kc2p2"
Nov 03 18:13:56.399044 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:56.399006    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c1b2d2-6557dc56cf-kc2p2"]
Nov 03 18:13:56.483967 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:56.476384    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-c1b2d2\""
Nov 03 18:13:56.507071 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:56.507037    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c1b2d2-7c79954dcf-7htl8"]
Nov 03 18:13:56.520753 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:56.520713    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d12d9244-eac0-4c66-b63d-14147ef13e3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c1b2d2-6557dc56cf-kc2p2\" (UID: \"d12d9244-eac0-4c66-b63d-14147ef13e3e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c1b2d2-6557dc56cf-kc2p2"
Nov 03 18:13:56.521488 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:56.521448    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s69lv\" (UniqueName: \"kubernetes.io/projected/d12d9244-eac0-4c66-b63d-14147ef13e3e-kube-api-access-s69lv\") pod \"payload-pre-processing-e2e-shared-b-w2-c1b2d2-6557dc56cf-kc2p2\" (UID: \"d12d9244-eac0-4c66-b63d-14147ef13e3e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c1b2d2-6557dc56cf-kc2p2"
Nov 03 18:13:56.704363 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:56.700320    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d12d9244-eac0-4c66-b63d-14147ef13e3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c1b2d2-6557dc56cf-kc2p2\" (UID: \"d12d9244-eac0-4c66-b63d-14147ef13e3e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c1b2d2-6557dc56cf-kc2p2"
Nov 03 18:13:56.706396 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:56.701465    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d12d9244-eac0-4c66-b63d-14147ef13e3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c1b2d2-6557dc56cf-kc2p2\" (UID: \"d12d9244-eac0-4c66-b63d-14147ef13e3e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c1b2d2-6557dc56cf-kc2p2"
Nov 03 18:13:56.706640 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:56.706479    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s69lv\" (UniqueName: \"kubernetes.io/projected/d12d9244-eac0-4c66-b63d-14147ef13e3e-kube-api-access-s69lv\") pod \"payload-pre-processing-e2e-shared-b-w2-c1b2d2-6557dc56cf-kc2p2\" (UID: \"d12d9244-eac0-4c66-b63d-14147ef13e3e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c1b2d2-6557dc56cf-kc2p2"
Nov 03 18:13:56.716968 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:56.711199    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c1b2d2-7c79954dcf-7htl8"]
Nov 03 18:13:56.716968 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:56.711315    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c1b2d2-7c79954dcf-7htl8"
Nov 03 18:13:56.760857 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:56.760828    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-c1b2d2-dockercfg-fqdb2\""
Nov 03 18:13:56.764177 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:56.764150    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s69lv\" (UniqueName: \"kubernetes.io/projected/d12d9244-eac0-4c66-b63d-14147ef13e3e-kube-api-access-s69lv\") pod \"payload-pre-processing-e2e-shared-b-w2-c1b2d2-6557dc56cf-kc2p2\" (UID: \"d12d9244-eac0-4c66-b63d-14147ef13e3e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c1b2d2-6557dc56cf-kc2p2"
Nov 03 18:13:56.789693 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:56.789604    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c1b2d2-6557dc56cf-kc2p2"
Nov 03 18:13:56.808764 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:56.808735    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3801739a-23aa-4cb1-9de5-b1aa0851fa93-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c1b2d2-7c79954dcf-7htl8\" (UID: \"3801739a-23aa-4cb1-9de5-b1aa0851fa93\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c1b2d2-7c79954dcf-7htl8"
Nov 03 18:13:56.808915 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:56.808857    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl59j\" (UniqueName: \"kubernetes.io/projected/3801739a-23aa-4cb1-9de5-b1aa0851fa93-kube-api-access-zl59j\") pod \"payload-processing-e2e-shared-b-w2-c1b2d2-7c79954dcf-7htl8\" (UID: \"3801739a-23aa-4cb1-9de5-b1aa0851fa93\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c1b2d2-7c79954dcf-7htl8"
Nov 03 18:13:56.916814 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:56.916769    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl59j\" (UniqueName: \"kubernetes.io/projected/3801739a-23aa-4cb1-9de5-b1aa0851fa93-kube-api-access-zl59j\") pod \"payload-processing-e2e-shared-b-w2-c1b2d2-7c79954dcf-7htl8\" (UID: \"3801739a-23aa-4cb1-9de5-b1aa0851fa93\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c1b2d2-7c79954dcf-7htl8"
Nov 03 18:13:56.917101 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:56.917074    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3801739a-23aa-4cb1-9de5-b1aa0851fa93-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c1b2d2-7c79954dcf-7htl8\" (UID: \"3801739a-23aa-4cb1-9de5-b1aa0851fa93\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c1b2d2-7c79954dcf-7htl8"
Nov 03 18:13:56.918194 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:56.918165    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3801739a-23aa-4cb1-9de5-b1aa0851fa93-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c1b2d2-7c79954dcf-7htl8\" (UID: \"3801739a-23aa-4cb1-9de5-b1aa0851fa93\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c1b2d2-7c79954dcf-7htl8"
Nov 03 18:13:56.932472 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:56.926423    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0adcbc-g486h"]
Nov 03 18:13:56.938358 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:56.938335    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0adcbc-g486h"
Nov 03 18:13:56.975762 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:56.975573    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl59j\" (UniqueName: \"kubernetes.io/projected/3801739a-23aa-4cb1-9de5-b1aa0851fa93-kube-api-access-zl59j\") pod \"payload-processing-e2e-shared-b-w2-c1b2d2-7c79954dcf-7htl8\" (UID: \"3801739a-23aa-4cb1-9de5-b1aa0851fa93\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c1b2d2-7c79954dcf-7htl8"
Nov 03 18:13:56.988392 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:56.988350    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0adcbc-g486h"]
Nov 03 18:13:56.988572 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:56.988395    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:13:56.988572 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:56.988368    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:13:56.988765 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:56.988603    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:13:56.988943 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:56.988923    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:13:57.018042 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:57.018002    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af71653e-b4ad-48a1-8ba9-bc8c1b10c44c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-0adcbc-g486h\" (UID: \"af71653e-b4ad-48a1-8ba9-bc8c1b10c44c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0adcbc-g486h"
Nov 03 18:13:57.018042 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:57.018045    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdvbg\" (UniqueName: \"kubernetes.io/projected/af71653e-b4ad-48a1-8ba9-bc8c1b10c44c-kube-api-access-jdvbg\") pod \"maas-api-revoke-keys-e2e-worker-w3-0adcbc-g486h\" (UID: \"af71653e-b4ad-48a1-8ba9-bc8c1b10c44c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0adcbc-g486h"
Nov 03 18:13:57.018296 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:57.018138    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af71653e-b4ad-48a1-8ba9-bc8c1b10c44c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-0adcbc-g486h\" (UID: \"af71653e-b4ad-48a1-8ba9-bc8c1b10c44c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0adcbc-g486h"
Nov 03 18:13:57.041114 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:57.041078    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c1b2d2-7c79954dcf-7htl8"
Nov 03 18:13:57.136382 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:57.132764    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af71653e-b4ad-48a1-8ba9-bc8c1b10c44c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-0adcbc-g486h\" (UID: \"af71653e-b4ad-48a1-8ba9-bc8c1b10c44c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0adcbc-g486h"
Nov 03 18:13:57.136382 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:57.132845    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdvbg\" (UniqueName: \"kubernetes.io/projected/af71653e-b4ad-48a1-8ba9-bc8c1b10c44c-kube-api-access-jdvbg\") pod \"maas-api-revoke-keys-e2e-worker-w3-0adcbc-g486h\" (UID: \"af71653e-b4ad-48a1-8ba9-bc8c1b10c44c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0adcbc-g486h"
Nov 03 18:13:57.136382 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:57.133014    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af71653e-b4ad-48a1-8ba9-bc8c1b10c44c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-0adcbc-g486h\" (UID: \"af71653e-b4ad-48a1-8ba9-bc8c1b10c44c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0adcbc-g486h"
Nov 03 18:13:57.136382 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:57.134031    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af71653e-b4ad-48a1-8ba9-bc8c1b10c44c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-0adcbc-g486h\" (UID: \"af71653e-b4ad-48a1-8ba9-bc8c1b10c44c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0adcbc-g486h"
Nov 03 18:13:57.137950 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:57.137191    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af71653e-b4ad-48a1-8ba9-bc8c1b10c44c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-0adcbc-g486h\" (UID: \"af71653e-b4ad-48a1-8ba9-bc8c1b10c44c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0adcbc-g486h"
Nov 03 18:13:57.181112 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:13:57.181072    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd12d9244_eac0_4c66_b63d_14147ef13e3e.slice/crio-02d78d8270ec880a2f2fa960c2c6ef05c8e426f092acd737c0f50fa14ceb8de1 WatchSource:0}: Error finding container 02d78d8270ec880a2f2fa960c2c6ef05c8e426f092acd737c0f50fa14ceb8de1: Status 404 returned error can't find the container with id 02d78d8270ec880a2f2fa960c2c6ef05c8e426f092acd737c0f50fa14ceb8de1
Nov 03 18:13:57.181802 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:57.181773    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c1b2d2-6557dc56cf-kc2p2"]
Nov 03 18:13:57.192472 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:57.192438    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdvbg\" (UniqueName: \"kubernetes.io/projected/af71653e-b4ad-48a1-8ba9-bc8c1b10c44c-kube-api-access-jdvbg\") pod \"maas-api-revoke-keys-e2e-worker-w3-0adcbc-g486h\" (UID: \"af71653e-b4ad-48a1-8ba9-bc8c1b10c44c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0adcbc-g486h"
Nov 03 18:13:57.261957 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:57.261925    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0adcbc-g486h"
Nov 03 18:13:57.424135 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:57.424094    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c1b2d2-7c79954dcf-7htl8"]
Nov 03 18:13:57.429514 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:13:57.429484    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3801739a_23aa_4cb1_9de5_b1aa0851fa93.slice/crio-0626fbf37c0368a47db28849fe9535c5fa5fc411feeef51ce53e98178bb4056d WatchSource:0}: Error finding container 0626fbf37c0368a47db28849fe9535c5fa5fc411feeef51ce53e98178bb4056d: Status 404 returned error can't find the container with id 0626fbf37c0368a47db28849fe9535c5fa5fc411feeef51ce53e98178bb4056d
Nov 03 18:13:57.639742 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:57.639692    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0adcbc-g486h"]
Nov 03 18:13:57.645012 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:13:57.644986    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf71653e_b4ad_48a1_8ba9_bc8c1b10c44c.slice/crio-efacf8ecb9b922ca2c3b37868a32bc90ee986fe70e8423dd108277ee42f145f3 WatchSource:0}: Error finding container efacf8ecb9b922ca2c3b37868a32bc90ee986fe70e8423dd108277ee42f145f3: Status 404 returned error can't find the container with id efacf8ecb9b922ca2c3b37868a32bc90ee986fe70e8423dd108277ee42f145f3
Nov 03 18:13:57.899927 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:57.899892    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0adcbc-g486h" event={"ID":"af71653e-b4ad-48a1-8ba9-bc8c1b10c44c","Type":"ContainerStarted","Data":"efacf8ecb9b922ca2c3b37868a32bc90ee986fe70e8423dd108277ee42f145f3"}
Nov 03 18:13:57.901447 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:57.901414    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c1b2d2-7c79954dcf-7htl8" event={"ID":"3801739a-23aa-4cb1-9de5-b1aa0851fa93","Type":"ContainerStarted","Data":"145f57e9ab112665d43e09123ffbf9aed0b8be18f9ffeb5bcc02010b79fc6a78"}
Nov 03 18:13:57.901590 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:57.901467    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c1b2d2-7c79954dcf-7htl8" event={"ID":"3801739a-23aa-4cb1-9de5-b1aa0851fa93","Type":"ContainerStarted","Data":"0626fbf37c0368a47db28849fe9535c5fa5fc411feeef51ce53e98178bb4056d"}
Nov 03 18:13:57.901590 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:57.901525    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c1b2d2-7c79954dcf-7htl8"
Nov 03 18:13:57.903360 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:57.903328    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c1b2d2-6557dc56cf-kc2p2" event={"ID":"d12d9244-eac0-4c66-b63d-14147ef13e3e","Type":"ContainerStarted","Data":"b56d3e3f7162e8af080dfca48d0cc4fa85eedcd629229632e3a2971fd101b13e"}
Nov 03 18:13:57.903486 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:57.903367    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c1b2d2-6557dc56cf-kc2p2" event={"ID":"d12d9244-eac0-4c66-b63d-14147ef13e3e","Type":"ContainerStarted","Data":"02d78d8270ec880a2f2fa960c2c6ef05c8e426f092acd737c0f50fa14ceb8de1"}
Nov 03 18:13:57.903486 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:57.903473    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c1b2d2-6557dc56cf-kc2p2"
Nov 03 18:13:58.014421 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:58.014365    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c1b2d2-7c79954dcf-7htl8" podStartSLOduration=2.014352154 podStartE2EDuration="2.014352154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:13:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:13:58.011060457 +0000 UTC m=+1677.027258527" watchObservedRunningTime="2026-11-03 18:13:58.014352154 +0000 UTC m=+1677.030550248"
Nov 03 18:13:58.121937 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:58.121827    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c1b2d2-6557dc56cf-kc2p2" podStartSLOduration=2.121811544 podStartE2EDuration="2.121811544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:13:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:13:58.118497973 +0000 UTC m=+1677.134696042" watchObservedRunningTime="2026-11-03 18:13:58.121811544 +0000 UTC m=+1677.138009618"
Nov 03 18:13:58.913471 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:58.913367    5315 generic.go:356] "Generic (PLEG): container finished" podID="af71653e-b4ad-48a1-8ba9-bc8c1b10c44c" containerID="d9cebf1e4e089f884819ad7303ec8ac0333f947756864677b1cfc1c47eebd260" exitCode=0
Nov 03 18:13:58.913937 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:58.913445    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0adcbc-g486h" event={"ID":"af71653e-b4ad-48a1-8ba9-bc8c1b10c44c","Type":"ContainerDied","Data":"d9cebf1e4e089f884819ad7303ec8ac0333f947756864677b1cfc1c47eebd260"}
Nov 03 18:13:59.645309 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:59.645271    5315 patch_prober.go:29] interesting pod/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:13:59.645482 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:13:59.645328    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w" podUID="f85c57a2-8cb8-4403-ba55-1de5bf63a921" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:14:00.102518 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:00.102479    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0adcbc-g486h"
Nov 03 18:14:00.188243 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:00.188156    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af71653e-b4ad-48a1-8ba9-bc8c1b10c44c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af71653e-b4ad-48a1-8ba9-bc8c1b10c44c-maas-api-openshift-ca\") pod \"af71653e-b4ad-48a1-8ba9-bc8c1b10c44c\" (UID: \"af71653e-b4ad-48a1-8ba9-bc8c1b10c44c\") "
Nov 03 18:14:00.188426 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:00.188262    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af71653e-b4ad-48a1-8ba9-bc8c1b10c44c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af71653e-b4ad-48a1-8ba9-bc8c1b10c44c-maas-api-xks-ca\") pod \"af71653e-b4ad-48a1-8ba9-bc8c1b10c44c\" (UID: \"af71653e-b4ad-48a1-8ba9-bc8c1b10c44c\") "
Nov 03 18:14:00.188426 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:00.188377    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af71653e-b4ad-48a1-8ba9-bc8c1b10c44c-kube-api-access-jdvbg\" (UniqueName: \"kubernetes.io/projected/af71653e-b4ad-48a1-8ba9-bc8c1b10c44c-kube-api-access-jdvbg\") pod \"af71653e-b4ad-48a1-8ba9-bc8c1b10c44c\" (UID: \"af71653e-b4ad-48a1-8ba9-bc8c1b10c44c\") "
Nov 03 18:14:00.188536 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:00.188466    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af71653e-b4ad-48a1-8ba9-bc8c1b10c44c-maas-api-openshift-ca" pod "af71653e-b4ad-48a1-8ba9-bc8c1b10c44c" (UID: "af71653e-b4ad-48a1-8ba9-bc8c1b10c44c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:14:00.189278 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:00.189256    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af71653e-b4ad-48a1-8ba9-bc8c1b10c44c-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:00.190483 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:00.190460    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af71653e-b4ad-48a1-8ba9-bc8c1b10c44c-maas-api-xks-ca" pod "af71653e-b4ad-48a1-8ba9-bc8c1b10c44c" (UID: "af71653e-b4ad-48a1-8ba9-bc8c1b10c44c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:14:00.190539 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:00.190508    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af71653e-b4ad-48a1-8ba9-bc8c1b10c44c-kube-api-access-jdvbg" pod "af71653e-b4ad-48a1-8ba9-bc8c1b10c44c" (UID: "af71653e-b4ad-48a1-8ba9-bc8c1b10c44c"). InnerVolumeSpecName "kube-api-access-jdvbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:00.290738 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:00.290690    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdvbg\" (UniqueName: \"kubernetes.io/projected/af71653e-b4ad-48a1-8ba9-bc8c1b10c44c-kube-api-access-jdvbg\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:00.290738 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:00.290732    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af71653e-b4ad-48a1-8ba9-bc8c1b10c44c-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:00.365545 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:00.365503    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn"]
Nov 03 18:14:00.365804 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:00.365776    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn" podUID="ab9a094b-19e7-4bf3-a4f6-63b953d47af2" containerName="istio-proxy" containerID="cri-o://6cfdec0dcdfd25265b87646dbbec47bf9ca3e9f2565de155f2e66c19c93ab698" gracePeriod=30
Nov 03 18:14:00.640060 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:00.640021    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67"]
Nov 03 18:14:00.640339 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:00.640307    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67" podUID="d7899300-736f-4a03-aef7-253d7a1b1e2f" containerName="istio-proxy" containerID="cri-o://ee4cb352ab64ce12813538b6442b423954bd6f3bb1ac379a481de909af752543" gracePeriod=30
Nov 03 18:14:01.009654 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:00.996849    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w"
Nov 03 18:14:01.019871 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.016633    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0adcbc-g486h" event={"ID":"af71653e-b4ad-48a1-8ba9-bc8c1b10c44c","Type":"ContainerDied","Data":"efacf8ecb9b922ca2c3b37868a32bc90ee986fe70e8423dd108277ee42f145f3"}
Nov 03 18:14:01.020226 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.020198    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efacf8ecb9b922ca2c3b37868a32bc90ee986fe70e8423dd108277ee42f145f3"
Nov 03 18:14:01.020393 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.017883    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0adcbc-g486h"
Nov 03 18:14:01.025664 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.025633    5315 generic.go:356] "Generic (PLEG): container finished" podID="f85c57a2-8cb8-4403-ba55-1de5bf63a921" containerID="174a67c0fd81c58870ea0e9570ac58ddb00de44b552e94daab9ad7c2c61cf31a" exitCode=0
Nov 03 18:14:01.026437 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.026161    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w"
Nov 03 18:14:01.026767 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.026183    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w" event={"ID":"f85c57a2-8cb8-4403-ba55-1de5bf63a921","Type":"ContainerDied","Data":"174a67c0fd81c58870ea0e9570ac58ddb00de44b552e94daab9ad7c2c61cf31a"}
Nov 03 18:14:01.026913 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.026897    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w" event={"ID":"f85c57a2-8cb8-4403-ba55-1de5bf63a921","Type":"ContainerDied","Data":"f39d5c3852ee329e20e20e7638cd38fb7ad176c288235b355d3ee2a063df0bdd"}
Nov 03 18:14:01.027019 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.027008    5315 scope.go:122] "RemoveContainer" containerID="174a67c0fd81c58870ea0e9570ac58ddb00de44b552e94daab9ad7c2c61cf31a"
Nov 03 18:14:01.036927 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.036891    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f85c57a2-8cb8-4403-ba55-1de5bf63a921-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f85c57a2-8cb8-4403-ba55-1de5bf63a921-istio-data\") pod \"f85c57a2-8cb8-4403-ba55-1de5bf63a921\" (UID: \"f85c57a2-8cb8-4403-ba55-1de5bf63a921\") "
Nov 03 18:14:01.037050 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.036985    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f85c57a2-8cb8-4403-ba55-1de5bf63a921-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f85c57a2-8cb8-4403-ba55-1de5bf63a921-istiod-ca-cert\") pod \"f85c57a2-8cb8-4403-ba55-1de5bf63a921\" (UID: \"f85c57a2-8cb8-4403-ba55-1de5bf63a921\") "
Nov 03 18:14:01.037123 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.037093    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f85c57a2-8cb8-4403-ba55-1de5bf63a921-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85c57a2-8cb8-4403-ba55-1de5bf63a921-workload-socket\") pod \"f85c57a2-8cb8-4403-ba55-1de5bf63a921\" (UID: \"f85c57a2-8cb8-4403-ba55-1de5bf63a921\") "
Nov 03 18:14:01.037181 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.037167    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f85c57a2-8cb8-4403-ba55-1de5bf63a921-istio-token\" (UniqueName: \"kubernetes.io/projected/f85c57a2-8cb8-4403-ba55-1de5bf63a921-istio-token\") pod \"f85c57a2-8cb8-4403-ba55-1de5bf63a921\" (UID: \"f85c57a2-8cb8-4403-ba55-1de5bf63a921\") "
Nov 03 18:14:01.037285 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.037266    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f85c57a2-8cb8-4403-ba55-1de5bf63a921-kube-api-access-xsdqn\" (UniqueName: \"kubernetes.io/projected/f85c57a2-8cb8-4403-ba55-1de5bf63a921-kube-api-access-xsdqn\") pod \"f85c57a2-8cb8-4403-ba55-1de5bf63a921\" (UID: \"f85c57a2-8cb8-4403-ba55-1de5bf63a921\") "
Nov 03 18:14:01.037355 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.037325    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f85c57a2-8cb8-4403-ba55-1de5bf63a921-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f85c57a2-8cb8-4403-ba55-1de5bf63a921-istio-podinfo\") pod \"f85c57a2-8cb8-4403-ba55-1de5bf63a921\" (UID: \"f85c57a2-8cb8-4403-ba55-1de5bf63a921\") "
Nov 03 18:14:01.037427 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.037408    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f85c57a2-8cb8-4403-ba55-1de5bf63a921-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85c57a2-8cb8-4403-ba55-1de5bf63a921-credential-socket\") pod \"f85c57a2-8cb8-4403-ba55-1de5bf63a921\" (UID: \"f85c57a2-8cb8-4403-ba55-1de5bf63a921\") "
Nov 03 18:14:01.037535 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.037517    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f85c57a2-8cb8-4403-ba55-1de5bf63a921-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f85c57a2-8cb8-4403-ba55-1de5bf63a921-istio-envoy\") pod \"f85c57a2-8cb8-4403-ba55-1de5bf63a921\" (UID: \"f85c57a2-8cb8-4403-ba55-1de5bf63a921\") "
Nov 03 18:14:01.037590 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.037562    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f85c57a2-8cb8-4403-ba55-1de5bf63a921-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f85c57a2-8cb8-4403-ba55-1de5bf63a921-workload-certs\") pod \"f85c57a2-8cb8-4403-ba55-1de5bf63a921\" (UID: \"f85c57a2-8cb8-4403-ba55-1de5bf63a921\") "
Nov 03 18:14:01.038298 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.037559    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f85c57a2-8cb8-4403-ba55-1de5bf63a921-istiod-ca-cert" pod "f85c57a2-8cb8-4403-ba55-1de5bf63a921" (UID: "f85c57a2-8cb8-4403-ba55-1de5bf63a921"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:14:01.038648 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.038594    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f85c57a2-8cb8-4403-ba55-1de5bf63a921-workload-socket" pod "f85c57a2-8cb8-4403-ba55-1de5bf63a921" (UID: "f85c57a2-8cb8-4403-ba55-1de5bf63a921"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:01.040942 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.040915    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f85c57a2-8cb8-4403-ba55-1de5bf63a921-istio-token" pod "f85c57a2-8cb8-4403-ba55-1de5bf63a921" (UID: "f85c57a2-8cb8-4403-ba55-1de5bf63a921"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:01.050760 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.047031    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f85c57a2-8cb8-4403-ba55-1de5bf63a921-workload-certs" pod "f85c57a2-8cb8-4403-ba55-1de5bf63a921" (UID: "f85c57a2-8cb8-4403-ba55-1de5bf63a921"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:01.050760 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.048684    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f85c57a2-8cb8-4403-ba55-1de5bf63a921-istio-data" pod "f85c57a2-8cb8-4403-ba55-1de5bf63a921" (UID: "f85c57a2-8cb8-4403-ba55-1de5bf63a921"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:01.050760 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.050697    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f85c57a2-8cb8-4403-ba55-1de5bf63a921-credential-socket" pod "f85c57a2-8cb8-4403-ba55-1de5bf63a921" (UID: "f85c57a2-8cb8-4403-ba55-1de5bf63a921"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:01.054435 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.054412    5315 scope.go:122] "RemoveContainer" containerID="095a31f3667a5bef8c64a5ae510403143a89669654e53377a42100159f0dd681"
Nov 03 18:14:01.057138 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.057104    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f85c57a2-8cb8-4403-ba55-1de5bf63a921-kube-api-access-xsdqn" pod "f85c57a2-8cb8-4403-ba55-1de5bf63a921" (UID: "f85c57a2-8cb8-4403-ba55-1de5bf63a921"). InnerVolumeSpecName "kube-api-access-xsdqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:01.065324 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.065305    5315 scope.go:122] "RemoveContainer" containerID="174a67c0fd81c58870ea0e9570ac58ddb00de44b552e94daab9ad7c2c61cf31a"
Nov 03 18:14:01.065672 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:14:01.065651    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"174a67c0fd81c58870ea0e9570ac58ddb00de44b552e94daab9ad7c2c61cf31a\": container with ID starting with 174a67c0fd81c58870ea0e9570ac58ddb00de44b552e94daab9ad7c2c61cf31a not found: ID does not exist" containerID="174a67c0fd81c58870ea0e9570ac58ddb00de44b552e94daab9ad7c2c61cf31a"
Nov 03 18:14:01.065728 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.065683    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"174a67c0fd81c58870ea0e9570ac58ddb00de44b552e94daab9ad7c2c61cf31a"} err="failed to get container status \"174a67c0fd81c58870ea0e9570ac58ddb00de44b552e94daab9ad7c2c61cf31a\": rpc error: code = NotFound desc = could not find container \"174a67c0fd81c58870ea0e9570ac58ddb00de44b552e94daab9ad7c2c61cf31a\": container with ID starting with 174a67c0fd81c58870ea0e9570ac58ddb00de44b552e94daab9ad7c2c61cf31a not found: ID does not exist"
Nov 03 18:14:01.065728 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.065706    5315 scope.go:122] "RemoveContainer" containerID="095a31f3667a5bef8c64a5ae510403143a89669654e53377a42100159f0dd681"
Nov 03 18:14:01.066013 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:14:01.065990    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"095a31f3667a5bef8c64a5ae510403143a89669654e53377a42100159f0dd681\": container with ID starting with 095a31f3667a5bef8c64a5ae510403143a89669654e53377a42100159f0dd681 not found: ID does not exist" containerID="095a31f3667a5bef8c64a5ae510403143a89669654e53377a42100159f0dd681"
Nov 03 18:14:01.066098 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.066019    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"095a31f3667a5bef8c64a5ae510403143a89669654e53377a42100159f0dd681"} err="failed to get container status \"095a31f3667a5bef8c64a5ae510403143a89669654e53377a42100159f0dd681\": rpc error: code = NotFound desc = could not find container \"095a31f3667a5bef8c64a5ae510403143a89669654e53377a42100159f0dd681\": container with ID starting with 095a31f3667a5bef8c64a5ae510403143a89669654e53377a42100159f0dd681 not found: ID does not exist"
Nov 03 18:14:01.070010 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.069987    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f85c57a2-8cb8-4403-ba55-1de5bf63a921-istio-podinfo" pod "f85c57a2-8cb8-4403-ba55-1de5bf63a921" (UID: "f85c57a2-8cb8-4403-ba55-1de5bf63a921"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:14:01.070088 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.069992    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f85c57a2-8cb8-4403-ba55-1de5bf63a921-istio-envoy" pod "f85c57a2-8cb8-4403-ba55-1de5bf63a921" (UID: "f85c57a2-8cb8-4403-ba55-1de5bf63a921"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:01.140651 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.140193    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f85c57a2-8cb8-4403-ba55-1de5bf63a921-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:01.140651 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.140224    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsdqn\" (UniqueName: \"kubernetes.io/projected/f85c57a2-8cb8-4403-ba55-1de5bf63a921-kube-api-access-xsdqn\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:01.140651 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.140235    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f85c57a2-8cb8-4403-ba55-1de5bf63a921-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:01.140651 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.140244    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85c57a2-8cb8-4403-ba55-1de5bf63a921-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:01.140651 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.140252    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f85c57a2-8cb8-4403-ba55-1de5bf63a921-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:01.140651 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.140260    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f85c57a2-8cb8-4403-ba55-1de5bf63a921-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:01.140651 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.140268    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f85c57a2-8cb8-4403-ba55-1de5bf63a921-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:01.140651 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.140278    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f85c57a2-8cb8-4403-ba55-1de5bf63a921-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:01.140651 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.140286    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85c57a2-8cb8-4403-ba55-1de5bf63a921-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:01.371188 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.371151    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7"]
Nov 03 18:14:01.452453 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.452035    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w"]
Nov 03 18:14:01.457140 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.457103    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b6cb8403-openshift-default-8646ddc5b5-jxx7w"]
Nov 03 18:14:01.579974 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.579935    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-260dd0-g6x4q"]
Nov 03 18:14:01.583140 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.583093    5315 cpu_manager.go:414] "containerMap: removing container" podUID="af71653e-b4ad-48a1-8ba9-bc8c1b10c44c" containerName="revoke-keys"
Nov 03 18:14:01.583140 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.583138    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af71653e-b4ad-48a1-8ba9-bc8c1b10c44c" containerName="revoke-keys"
Nov 03 18:14:01.583375 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.583211    5315 cpu_manager.go:414] "containerMap: removing container" podUID="f85c57a2-8cb8-4403-ba55-1de5bf63a921" containerName="istio-proxy"
Nov 03 18:14:01.583375 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.583220    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f85c57a2-8cb8-4403-ba55-1de5bf63a921" containerName="istio-proxy"
Nov 03 18:14:01.583375 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.583235    5315 cpu_manager.go:414] "containerMap: removing container" podUID="f85c57a2-8cb8-4403-ba55-1de5bf63a921" containerName="istio-proxy"
Nov 03 18:14:01.583375 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.583242    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f85c57a2-8cb8-4403-ba55-1de5bf63a921" containerName="istio-proxy"
Nov 03 18:14:01.583622 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.583589    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="f85c57a2-8cb8-4403-ba55-1de5bf63a921" containerName="istio-proxy"
Nov 03 18:14:01.583693 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.583627    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="f85c57a2-8cb8-4403-ba55-1de5bf63a921" containerName="istio-proxy"
Nov 03 18:14:01.583693 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.583664    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="af71653e-b4ad-48a1-8ba9-bc8c1b10c44c" containerName="revoke-keys"
Nov 03 18:14:01.593893 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.593853    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-260dd0-g6x4q"
Nov 03 18:14:01.640142 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.640048    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-260dd0-g6x4q"]
Nov 03 18:14:01.642636 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.642583    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:14:01.642895 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.642843    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:14:01.643178 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.643147    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:14:01.643727 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.643706    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:14:01.646780 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.646747    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9322d0f4-e8ca-483a-894e-62491681581c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-260dd0-g6x4q\" (UID: \"9322d0f4-e8ca-483a-894e-62491681581c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-260dd0-g6x4q"
Nov 03 18:14:01.646912 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.646787    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9322d0f4-e8ca-483a-894e-62491681581c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-260dd0-g6x4q\" (UID: \"9322d0f4-e8ca-483a-894e-62491681581c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-260dd0-g6x4q"
Nov 03 18:14:01.646980 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.646927    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j4vr\" (UniqueName: \"kubernetes.io/projected/9322d0f4-e8ca-483a-894e-62491681581c-kube-api-access-8j4vr\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-260dd0-g6x4q\" (UID: \"9322d0f4-e8ca-483a-894e-62491681581c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-260dd0-g6x4q"
Nov 03 18:14:01.748602 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.748565    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9322d0f4-e8ca-483a-894e-62491681581c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-260dd0-g6x4q\" (UID: \"9322d0f4-e8ca-483a-894e-62491681581c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-260dd0-g6x4q"
Nov 03 18:14:01.748602 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.748605    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9322d0f4-e8ca-483a-894e-62491681581c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-260dd0-g6x4q\" (UID: \"9322d0f4-e8ca-483a-894e-62491681581c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-260dd0-g6x4q"
Nov 03 18:14:01.748889 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.748693    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j4vr\" (UniqueName: \"kubernetes.io/projected/9322d0f4-e8ca-483a-894e-62491681581c-kube-api-access-8j4vr\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-260dd0-g6x4q\" (UID: \"9322d0f4-e8ca-483a-894e-62491681581c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-260dd0-g6x4q"
Nov 03 18:14:01.749314 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.749287    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9322d0f4-e8ca-483a-894e-62491681581c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-260dd0-g6x4q\" (UID: \"9322d0f4-e8ca-483a-894e-62491681581c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-260dd0-g6x4q"
Nov 03 18:14:01.751068 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.751048    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9322d0f4-e8ca-483a-894e-62491681581c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-260dd0-g6x4q\" (UID: \"9322d0f4-e8ca-483a-894e-62491681581c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-260dd0-g6x4q"
Nov 03 18:14:01.806537 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.806497    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j4vr\" (UniqueName: \"kubernetes.io/projected/9322d0f4-e8ca-483a-894e-62491681581c-kube-api-access-8j4vr\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-260dd0-g6x4q\" (UID: \"9322d0f4-e8ca-483a-894e-62491681581c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-260dd0-g6x4q"
Nov 03 18:14:01.908590 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:01.908502    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-260dd0-g6x4q"
Nov 03 18:14:02.286306 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:02.286264    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-260dd0-g6x4q"]
Nov 03 18:14:02.292011 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:14:02.291983    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9322d0f4_e8ca_483a_894e_62491681581c.slice/crio-76cdc12cb662eeacd4f8241016ab4d12f98e74bc81003672a5c1a45ddbce82e0 WatchSource:0}: Error finding container 76cdc12cb662eeacd4f8241016ab4d12f98e74bc81003672a5c1a45ddbce82e0: Status 404 returned error can't find the container with id 76cdc12cb662eeacd4f8241016ab4d12f98e74bc81003672a5c1a45ddbce82e0
Nov 03 18:14:02.669173 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:02.669133    5315 patch_prober.go:29] interesting pod/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:14:02.669327 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:02.669200    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67" podUID="d7899300-736f-4a03-aef7-253d7a1b1e2f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:14:03.046783 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:03.046744    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-260dd0-g6x4q" event={"ID":"9322d0f4-e8ca-483a-894e-62491681581c","Type":"ContainerStarted","Data":"93aec41d7cb9abdbae904ffea4b5cf4e6b61228c984e0ad21ef4fb53513b4677"}
Nov 03 18:14:03.046974 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:03.046797    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-260dd0-g6x4q" event={"ID":"9322d0f4-e8ca-483a-894e-62491681581c","Type":"ContainerStarted","Data":"76cdc12cb662eeacd4f8241016ab4d12f98e74bc81003672a5c1a45ddbce82e0"}
Nov 03 18:14:03.106693 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:03.106660    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f85c57a2-8cb8-4403-ba55-1de5bf63a921" path="/var/lib/kubelet/pods/f85c57a2-8cb8-4403-ba55-1de5bf63a921/volumes"
Nov 03 18:14:03.157062 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:03.157004    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-260dd0-g6x4q" podStartSLOduration=2.156991018 podStartE2EDuration="2.156991018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:14:03.155100656 +0000 UTC m=+1682.171298725" watchObservedRunningTime="2026-11-03 18:14:03.156991018 +0000 UTC m=+1682.173189087"
Nov 03 18:14:04.056558 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:04.056518    5315 generic.go:356] "Generic (PLEG): container finished" podID="9322d0f4-e8ca-483a-894e-62491681581c" containerID="93aec41d7cb9abdbae904ffea4b5cf4e6b61228c984e0ad21ef4fb53513b4677" exitCode=0
Nov 03 18:14:04.057023 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:04.056650    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-260dd0-g6x4q" event={"ID":"9322d0f4-e8ca-483a-894e-62491681581c","Type":"ContainerDied","Data":"93aec41d7cb9abdbae904ffea4b5cf4e6b61228c984e0ad21ef4fb53513b4677"}
Nov 03 18:14:04.401308 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:04.401213    5315 scope.go:122] "RemoveContainer" containerID="435362a644b1ee21b1ed74f3effb3cec52f9c2a1effc551b50d7c54542719e05"
Nov 03 18:14:04.414729 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:04.414702    5315 scope.go:122] "RemoveContainer" containerID="db8941ae2d8da604976cbe28d6e9658d8875a2106d9612ed4356380a82ece702"
Nov 03 18:14:04.425736 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:04.425706    5315 scope.go:122] "RemoveContainer" containerID="1898f53c3072938b45dd415a66459597fa9290d6524987c75ead57c10671034f"
Nov 03 18:14:04.508366 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:04.508330    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-0ccbb9-57dcc5ff7-zcv75" podUID="b86eb9eb-969a-4ac5-a947-0f1d32ca5b72" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:14:04.508366 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:04.508354    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0ccbb9-6b4b8ccc7f-q47rc" podUID="e7580e5e-cc8b-412e-99f0-3a71c762e618" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:14:04.851909 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:04.851867    5315 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:14:04.852114 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:04.851943    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn" podUID="ab9a094b-19e7-4bf3-a4f6-63b953d47af2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:14:05.232677 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:05.232649    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-260dd0-g6x4q"
Nov 03 18:14:05.307533 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:05.307499    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9322d0f4-e8ca-483a-894e-62491681581c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9322d0f4-e8ca-483a-894e-62491681581c-maas-api-xks-ca\") pod \"9322d0f4-e8ca-483a-894e-62491681581c\" (UID: \"9322d0f4-e8ca-483a-894e-62491681581c\") "
Nov 03 18:14:05.307731 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:05.307592    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9322d0f4-e8ca-483a-894e-62491681581c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9322d0f4-e8ca-483a-894e-62491681581c-maas-api-openshift-ca\") pod \"9322d0f4-e8ca-483a-894e-62491681581c\" (UID: \"9322d0f4-e8ca-483a-894e-62491681581c\") "
Nov 03 18:14:05.307931 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:05.307898    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9322d0f4-e8ca-483a-894e-62491681581c-kube-api-access-8j4vr\" (UniqueName: \"kubernetes.io/projected/9322d0f4-e8ca-483a-894e-62491681581c-kube-api-access-8j4vr\") pod \"9322d0f4-e8ca-483a-894e-62491681581c\" (UID: \"9322d0f4-e8ca-483a-894e-62491681581c\") "
Nov 03 18:14:05.308022 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:05.307980    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9322d0f4-e8ca-483a-894e-62491681581c-maas-api-openshift-ca" pod "9322d0f4-e8ca-483a-894e-62491681581c" (UID: "9322d0f4-e8ca-483a-894e-62491681581c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:14:05.308497 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:05.308472    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9322d0f4-e8ca-483a-894e-62491681581c-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:05.309827 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:05.309798    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9322d0f4-e8ca-483a-894e-62491681581c-maas-api-xks-ca" pod "9322d0f4-e8ca-483a-894e-62491681581c" (UID: "9322d0f4-e8ca-483a-894e-62491681581c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:14:05.309910 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:05.309822    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9322d0f4-e8ca-483a-894e-62491681581c-kube-api-access-8j4vr" pod "9322d0f4-e8ca-483a-894e-62491681581c" (UID: "9322d0f4-e8ca-483a-894e-62491681581c"). InnerVolumeSpecName "kube-api-access-8j4vr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:05.409553 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:05.409523    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j4vr\" (UniqueName: \"kubernetes.io/projected/9322d0f4-e8ca-483a-894e-62491681581c-kube-api-access-8j4vr\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:05.409553 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:05.409554    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9322d0f4-e8ca-483a-894e-62491681581c-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:05.612851 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:05.612824    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn"
Nov 03 18:14:05.713126 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:05.713090    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-workload-socket\") pod \"ab9a094b-19e7-4bf3-a4f6-63b953d47af2\" (UID: \"ab9a094b-19e7-4bf3-a4f6-63b953d47af2\") "
Nov 03 18:14:05.713308 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:05.713140    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-workload-certs\") pod \"ab9a094b-19e7-4bf3-a4f6-63b953d47af2\" (UID: \"ab9a094b-19e7-4bf3-a4f6-63b953d47af2\") "
Nov 03 18:14:05.713308 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:05.713183    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-istiod-ca-cert\") pod \"ab9a094b-19e7-4bf3-a4f6-63b953d47af2\" (UID: \"ab9a094b-19e7-4bf3-a4f6-63b953d47af2\") "
Nov 03 18:14:05.713308 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:05.713213    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-credential-socket\") pod \"ab9a094b-19e7-4bf3-a4f6-63b953d47af2\" (UID: \"ab9a094b-19e7-4bf3-a4f6-63b953d47af2\") "
Nov 03 18:14:05.713308 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:05.713255    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-kube-api-access-4kglx\" (UniqueName: \"kubernetes.io/projected/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-kube-api-access-4kglx\") pod \"ab9a094b-19e7-4bf3-a4f6-63b953d47af2\" (UID: \"ab9a094b-19e7-4bf3-a4f6-63b953d47af2\") "
Nov 03 18:14:05.713526 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:05.713366    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-istio-token\" (UniqueName: \"kubernetes.io/projected/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-istio-token\") pod \"ab9a094b-19e7-4bf3-a4f6-63b953d47af2\" (UID: \"ab9a094b-19e7-4bf3-a4f6-63b953d47af2\") "
Nov 03 18:14:05.713526 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:05.713395    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-istio-envoy\") pod \"ab9a094b-19e7-4bf3-a4f6-63b953d47af2\" (UID: \"ab9a094b-19e7-4bf3-a4f6-63b953d47af2\") "
Nov 03 18:14:05.713526 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:05.713406    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-workload-socket" pod "ab9a094b-19e7-4bf3-a4f6-63b953d47af2" (UID: "ab9a094b-19e7-4bf3-a4f6-63b953d47af2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:05.713526 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:05.713509    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-istio-podinfo\") pod \"ab9a094b-19e7-4bf3-a4f6-63b953d47af2\" (UID: \"ab9a094b-19e7-4bf3-a4f6-63b953d47af2\") "
Nov 03 18:14:05.713740 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:05.713518    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-workload-certs" pod "ab9a094b-19e7-4bf3-a4f6-63b953d47af2" (UID: "ab9a094b-19e7-4bf3-a4f6-63b953d47af2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:05.713740 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:05.713563    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-credential-socket" pod "ab9a094b-19e7-4bf3-a4f6-63b953d47af2" (UID: "ab9a094b-19e7-4bf3-a4f6-63b953d47af2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:05.713740 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:05.713643    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-istiod-ca-cert" pod "ab9a094b-19e7-4bf3-a4f6-63b953d47af2" (UID: "ab9a094b-19e7-4bf3-a4f6-63b953d47af2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:14:05.713740 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:05.713697    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-istio-data\") pod \"ab9a094b-19e7-4bf3-a4f6-63b953d47af2\" (UID: \"ab9a094b-19e7-4bf3-a4f6-63b953d47af2\") "
Nov 03 18:14:05.714095 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:05.714067    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-istio-data" pod "ab9a094b-19e7-4bf3-a4f6-63b953d47af2" (UID: "ab9a094b-19e7-4bf3-a4f6-63b953d47af2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:05.714473 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:05.714440    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:05.714473 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:05.714472    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:05.714705 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:05.714488    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:05.714705 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:05.714505    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:05.714705 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:05.714520    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:05.715903 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:05.715882    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-istio-podinfo" pod "ab9a094b-19e7-4bf3-a4f6-63b953d47af2" (UID: "ab9a094b-19e7-4bf3-a4f6-63b953d47af2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:14:05.716011 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:05.715996    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-istio-envoy" pod "ab9a094b-19e7-4bf3-a4f6-63b953d47af2" (UID: "ab9a094b-19e7-4bf3-a4f6-63b953d47af2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:05.716063 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:05.716014    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-kube-api-access-4kglx" pod "ab9a094b-19e7-4bf3-a4f6-63b953d47af2" (UID: "ab9a094b-19e7-4bf3-a4f6-63b953d47af2"). InnerVolumeSpecName "kube-api-access-4kglx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:05.716976 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:05.716959    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-istio-token" pod "ab9a094b-19e7-4bf3-a4f6-63b953d47af2" (UID: "ab9a094b-19e7-4bf3-a4f6-63b953d47af2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:05.815858 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:05.815821    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:05.815858 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:05.815859    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:05.816054 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:05.815874    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:05.816054 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:05.815889    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kglx\" (UniqueName: \"kubernetes.io/projected/ab9a094b-19e7-4bf3-a4f6-63b953d47af2-kube-api-access-4kglx\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:05.984633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:05.983742    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67"
Nov 03 18:14:06.120635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.120083    5315 generic.go:356] "Generic (PLEG): container finished" podID="ab9a094b-19e7-4bf3-a4f6-63b953d47af2" containerID="6cfdec0dcdfd25265b87646dbbec47bf9ca3e9f2565de155f2e66c19c93ab698" exitCode=0
Nov 03 18:14:06.120635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.120357    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn" event={"ID":"ab9a094b-19e7-4bf3-a4f6-63b953d47af2","Type":"ContainerDied","Data":"6cfdec0dcdfd25265b87646dbbec47bf9ca3e9f2565de155f2e66c19c93ab698"}
Nov 03 18:14:06.120635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.120412    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn" event={"ID":"ab9a094b-19e7-4bf3-a4f6-63b953d47af2","Type":"ContainerDied","Data":"a4a24619592032e4702259ceee01c196cbe87123c2024ffc39d959d6b84bd23e"}
Nov 03 18:14:06.120635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.120444    5315 scope.go:122] "RemoveContainer" containerID="6cfdec0dcdfd25265b87646dbbec47bf9ca3e9f2565de155f2e66c19c93ab698"
Nov 03 18:14:06.120635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.120645    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn"
Nov 03 18:14:06.152046 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.135458    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7899300-736f-4a03-aef7-253d7a1b1e2f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7899300-736f-4a03-aef7-253d7a1b1e2f-istio-data\") pod \"d7899300-736f-4a03-aef7-253d7a1b1e2f\" (UID: \"d7899300-736f-4a03-aef7-253d7a1b1e2f\") "
Nov 03 18:14:06.152046 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.135535    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7899300-736f-4a03-aef7-253d7a1b1e2f-kube-api-access-b2v2j\" (UniqueName: \"kubernetes.io/projected/d7899300-736f-4a03-aef7-253d7a1b1e2f-kube-api-access-b2v2j\") pod \"d7899300-736f-4a03-aef7-253d7a1b1e2f\" (UID: \"d7899300-736f-4a03-aef7-253d7a1b1e2f\") "
Nov 03 18:14:06.152046 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.135666    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7899300-736f-4a03-aef7-253d7a1b1e2f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7899300-736f-4a03-aef7-253d7a1b1e2f-credential-socket\") pod \"d7899300-736f-4a03-aef7-253d7a1b1e2f\" (UID: \"d7899300-736f-4a03-aef7-253d7a1b1e2f\") "
Nov 03 18:14:06.152046 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.135765    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7899300-736f-4a03-aef7-253d7a1b1e2f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7899300-736f-4a03-aef7-253d7a1b1e2f-istiod-ca-cert\") pod \"d7899300-736f-4a03-aef7-253d7a1b1e2f\" (UID: \"d7899300-736f-4a03-aef7-253d7a1b1e2f\") "
Nov 03 18:14:06.152046 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.135809    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7899300-736f-4a03-aef7-253d7a1b1e2f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7899300-736f-4a03-aef7-253d7a1b1e2f-workload-socket\") pod \"d7899300-736f-4a03-aef7-253d7a1b1e2f\" (UID: \"d7899300-736f-4a03-aef7-253d7a1b1e2f\") "
Nov 03 18:14:06.152046 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.135881    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7899300-736f-4a03-aef7-253d7a1b1e2f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7899300-736f-4a03-aef7-253d7a1b1e2f-workload-certs\") pod \"d7899300-736f-4a03-aef7-253d7a1b1e2f\" (UID: \"d7899300-736f-4a03-aef7-253d7a1b1e2f\") "
Nov 03 18:14:06.152046 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.135967    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7899300-736f-4a03-aef7-253d7a1b1e2f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7899300-736f-4a03-aef7-253d7a1b1e2f-istio-podinfo\") pod \"d7899300-736f-4a03-aef7-253d7a1b1e2f\" (UID: \"d7899300-736f-4a03-aef7-253d7a1b1e2f\") "
Nov 03 18:14:06.152046 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.136071    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7899300-736f-4a03-aef7-253d7a1b1e2f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7899300-736f-4a03-aef7-253d7a1b1e2f-istio-envoy\") pod \"d7899300-736f-4a03-aef7-253d7a1b1e2f\" (UID: \"d7899300-736f-4a03-aef7-253d7a1b1e2f\") "
Nov 03 18:14:06.152046 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.136145    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7899300-736f-4a03-aef7-253d7a1b1e2f-istio-token\" (UniqueName: \"kubernetes.io/projected/d7899300-736f-4a03-aef7-253d7a1b1e2f-istio-token\") pod \"d7899300-736f-4a03-aef7-253d7a1b1e2f\" (UID: \"d7899300-736f-4a03-aef7-253d7a1b1e2f\") "
Nov 03 18:14:06.152046 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.147353    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7899300-736f-4a03-aef7-253d7a1b1e2f-istiod-ca-cert" pod "d7899300-736f-4a03-aef7-253d7a1b1e2f" (UID: "d7899300-736f-4a03-aef7-253d7a1b1e2f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:14:06.152046 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.147703    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7899300-736f-4a03-aef7-253d7a1b1e2f-istio-data" pod "d7899300-736f-4a03-aef7-253d7a1b1e2f" (UID: "d7899300-736f-4a03-aef7-253d7a1b1e2f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:06.152046 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.148251    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7899300-736f-4a03-aef7-253d7a1b1e2f-credential-socket" pod "d7899300-736f-4a03-aef7-253d7a1b1e2f" (UID: "d7899300-736f-4a03-aef7-253d7a1b1e2f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:06.156306 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.154764    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7899300-736f-4a03-aef7-253d7a1b1e2f-istio-token" pod "d7899300-736f-4a03-aef7-253d7a1b1e2f" (UID: "d7899300-736f-4a03-aef7-253d7a1b1e2f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:06.156306 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.155018    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7899300-736f-4a03-aef7-253d7a1b1e2f-workload-socket" pod "d7899300-736f-4a03-aef7-253d7a1b1e2f" (UID: "d7899300-736f-4a03-aef7-253d7a1b1e2f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:06.156306 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.155192    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7899300-736f-4a03-aef7-253d7a1b1e2f-workload-certs" pod "d7899300-736f-4a03-aef7-253d7a1b1e2f" (UID: "d7899300-736f-4a03-aef7-253d7a1b1e2f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:06.161377 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.158417    5315 generic.go:356] "Generic (PLEG): container finished" podID="d7899300-736f-4a03-aef7-253d7a1b1e2f" containerID="ee4cb352ab64ce12813538b6442b423954bd6f3bb1ac379a481de909af752543" exitCode=0
Nov 03 18:14:06.161377 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.158601    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67" event={"ID":"d7899300-736f-4a03-aef7-253d7a1b1e2f","Type":"ContainerDied","Data":"ee4cb352ab64ce12813538b6442b423954bd6f3bb1ac379a481de909af752543"}
Nov 03 18:14:06.161377 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.158673    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67" event={"ID":"d7899300-736f-4a03-aef7-253d7a1b1e2f","Type":"ContainerDied","Data":"73e94ca374c06fe77e918bd7d4422d2f782937f3507b954b29b016ef43152e6d"}
Nov 03 18:14:06.161377 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.158754    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67"
Nov 03 18:14:06.193804 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.173902    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-260dd0-g6x4q" event={"ID":"9322d0f4-e8ca-483a-894e-62491681581c","Type":"ContainerDied","Data":"76cdc12cb662eeacd4f8241016ab4d12f98e74bc81003672a5c1a45ddbce82e0"}
Nov 03 18:14:06.193804 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.173974    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76cdc12cb662eeacd4f8241016ab4d12f98e74bc81003672a5c1a45ddbce82e0"
Nov 03 18:14:06.193804 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.174047    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-260dd0-g6x4q"
Nov 03 18:14:06.193804 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.179824    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7899300-736f-4a03-aef7-253d7a1b1e2f-istio-envoy" pod "d7899300-736f-4a03-aef7-253d7a1b1e2f" (UID: "d7899300-736f-4a03-aef7-253d7a1b1e2f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:06.193804 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.191029    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7899300-736f-4a03-aef7-253d7a1b1e2f-kube-api-access-b2v2j" pod "d7899300-736f-4a03-aef7-253d7a1b1e2f" (UID: "d7899300-736f-4a03-aef7-253d7a1b1e2f"). InnerVolumeSpecName "kube-api-access-b2v2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:06.193804 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.192333    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7899300-736f-4a03-aef7-253d7a1b1e2f-istio-podinfo" pod "d7899300-736f-4a03-aef7-253d7a1b1e2f" (UID: "d7899300-736f-4a03-aef7-253d7a1b1e2f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:14:06.207571 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.202328    5315 scope.go:122] "RemoveContainer" containerID="6cfdec0dcdfd25265b87646dbbec47bf9ca3e9f2565de155f2e66c19c93ab698"
Nov 03 18:14:06.208042 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:14:06.208014    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cfdec0dcdfd25265b87646dbbec47bf9ca3e9f2565de155f2e66c19c93ab698\": container with ID starting with 6cfdec0dcdfd25265b87646dbbec47bf9ca3e9f2565de155f2e66c19c93ab698 not found: ID does not exist" containerID="6cfdec0dcdfd25265b87646dbbec47bf9ca3e9f2565de155f2e66c19c93ab698"
Nov 03 18:14:06.208162 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.208053    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cfdec0dcdfd25265b87646dbbec47bf9ca3e9f2565de155f2e66c19c93ab698"} err="failed to get container status \"6cfdec0dcdfd25265b87646dbbec47bf9ca3e9f2565de155f2e66c19c93ab698\": rpc error: code = NotFound desc = could not find container \"6cfdec0dcdfd25265b87646dbbec47bf9ca3e9f2565de155f2e66c19c93ab698\": container with ID starting with 6cfdec0dcdfd25265b87646dbbec47bf9ca3e9f2565de155f2e66c19c93ab698 not found: ID does not exist"
Nov 03 18:14:06.208162 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.208081    5315 scope.go:122] "RemoveContainer" containerID="ee4cb352ab64ce12813538b6442b423954bd6f3bb1ac379a481de909af752543"
Nov 03 18:14:06.239045 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.239003    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7899300-736f-4a03-aef7-253d7a1b1e2f-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:06.239045 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.239047    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7899300-736f-4a03-aef7-253d7a1b1e2f-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:06.239593 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.239065    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7899300-736f-4a03-aef7-253d7a1b1e2f-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:06.239593 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.239081    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7899300-736f-4a03-aef7-253d7a1b1e2f-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:06.239593 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.239097    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7899300-736f-4a03-aef7-253d7a1b1e2f-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:06.239593 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.239110    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7899300-736f-4a03-aef7-253d7a1b1e2f-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:06.239593 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.239124    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7899300-736f-4a03-aef7-253d7a1b1e2f-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:06.239593 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.239137    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2v2j\" (UniqueName: \"kubernetes.io/projected/d7899300-736f-4a03-aef7-253d7a1b1e2f-kube-api-access-b2v2j\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:06.239593 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.239151    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7899300-736f-4a03-aef7-253d7a1b1e2f-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:06.240200 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.240177    5315 scope.go:122] "RemoveContainer" containerID="750e09020663ab70c545beeced3a22c7d61ce9e5431603a6d98aa87a8eeb4264"
Nov 03 18:14:06.251595 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.251563    5315 scope.go:122] "RemoveContainer" containerID="ee4cb352ab64ce12813538b6442b423954bd6f3bb1ac379a481de909af752543"
Nov 03 18:14:06.251894 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:14:06.251868    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee4cb352ab64ce12813538b6442b423954bd6f3bb1ac379a481de909af752543\": container with ID starting with ee4cb352ab64ce12813538b6442b423954bd6f3bb1ac379a481de909af752543 not found: ID does not exist" containerID="ee4cb352ab64ce12813538b6442b423954bd6f3bb1ac379a481de909af752543"
Nov 03 18:14:06.251984 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.251901    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee4cb352ab64ce12813538b6442b423954bd6f3bb1ac379a481de909af752543"} err="failed to get container status \"ee4cb352ab64ce12813538b6442b423954bd6f3bb1ac379a481de909af752543\": rpc error: code = NotFound desc = could not find container \"ee4cb352ab64ce12813538b6442b423954bd6f3bb1ac379a481de909af752543\": container with ID starting with ee4cb352ab64ce12813538b6442b423954bd6f3bb1ac379a481de909af752543 not found: ID does not exist"
Nov 03 18:14:06.251984 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.251925    5315 scope.go:122] "RemoveContainer" containerID="750e09020663ab70c545beeced3a22c7d61ce9e5431603a6d98aa87a8eeb4264"
Nov 03 18:14:06.252239 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:14:06.252209    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"750e09020663ab70c545beeced3a22c7d61ce9e5431603a6d98aa87a8eeb4264\": container with ID starting with 750e09020663ab70c545beeced3a22c7d61ce9e5431603a6d98aa87a8eeb4264 not found: ID does not exist" containerID="750e09020663ab70c545beeced3a22c7d61ce9e5431603a6d98aa87a8eeb4264"
Nov 03 18:14:06.252292 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.252251    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"750e09020663ab70c545beeced3a22c7d61ce9e5431603a6d98aa87a8eeb4264"} err="failed to get container status \"750e09020663ab70c545beeced3a22c7d61ce9e5431603a6d98aa87a8eeb4264\": rpc error: code = NotFound desc = could not find container \"750e09020663ab70c545beeced3a22c7d61ce9e5431603a6d98aa87a8eeb4264\": container with ID starting with 750e09020663ab70c545beeced3a22c7d61ce9e5431603a6d98aa87a8eeb4264 not found: ID does not exist"
Nov 03 18:14:06.338866 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.338829    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn"]
Nov 03 18:14:06.343092 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.343039    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-rrtrn"]
Nov 03 18:14:06.575746 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.575710    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67"]
Nov 03 18:14:06.581210 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:06.581177    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-c91bee-openshift-default-6b467cf69-7fj67"]
Nov 03 18:14:07.107908 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:07.107875    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab9a094b-19e7-4bf3-a4f6-63b953d47af2" path="/var/lib/kubelet/pods/ab9a094b-19e7-4bf3-a4f6-63b953d47af2/volumes"
Nov 03 18:14:07.108453 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:07.108437    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7899300-736f-4a03-aef7-253d7a1b1e2f" path="/var/lib/kubelet/pods/d7899300-736f-4a03-aef7-253d7a1b1e2f/volumes"
Nov 03 18:14:07.273634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:07.270695    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aab753-469nk"]
Nov 03 18:14:07.286639 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:07.280722    5315 cpu_manager.go:414] "containerMap: removing container" podUID="d7899300-736f-4a03-aef7-253d7a1b1e2f" containerName="istio-proxy"
Nov 03 18:14:07.286639 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:07.280758    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7899300-736f-4a03-aef7-253d7a1b1e2f" containerName="istio-proxy"
Nov 03 18:14:07.286639 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:07.280815    5315 cpu_manager.go:414] "containerMap: removing container" podUID="9322d0f4-e8ca-483a-894e-62491681581c" containerName="revoke-keys"
Nov 03 18:14:07.286639 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:07.280825    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9322d0f4-e8ca-483a-894e-62491681581c" containerName="revoke-keys"
Nov 03 18:14:07.286639 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:07.280862    5315 cpu_manager.go:414] "containerMap: removing container" podUID="ab9a094b-19e7-4bf3-a4f6-63b953d47af2" containerName="istio-proxy"
Nov 03 18:14:07.286639 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:07.280872    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab9a094b-19e7-4bf3-a4f6-63b953d47af2" containerName="istio-proxy"
Nov 03 18:14:07.286639 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:07.281098    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7899300-736f-4a03-aef7-253d7a1b1e2f" containerName="istio-proxy"
Nov 03 18:14:07.286639 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:07.281112    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7899300-736f-4a03-aef7-253d7a1b1e2f" containerName="istio-proxy"
Nov 03 18:14:07.286639 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:07.281127    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="9322d0f4-e8ca-483a-894e-62491681581c" containerName="revoke-keys"
Nov 03 18:14:07.286639 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:07.281139    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab9a094b-19e7-4bf3-a4f6-63b953d47af2" containerName="istio-proxy"
Nov 03 18:14:07.286639 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:07.285685    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aab753-469nk"
Nov 03 18:14:07.320017 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:07.319972    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aab753-469nk"]
Nov 03 18:14:07.356720 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:07.355176    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b15f8554-6d3e-46bf-886f-e32da257adcc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-aab753-469nk\" (UID: \"b15f8554-6d3e-46bf-886f-e32da257adcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aab753-469nk"
Nov 03 18:14:07.356720 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:07.355444    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhvzz\" (UniqueName: \"kubernetes.io/projected/b15f8554-6d3e-46bf-886f-e32da257adcc-kube-api-access-qhvzz\") pod \"maas-api-revoke-keys-e2e-worker-w6-aab753-469nk\" (UID: \"b15f8554-6d3e-46bf-886f-e32da257adcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aab753-469nk"
Nov 03 18:14:07.356720 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:07.355482    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b15f8554-6d3e-46bf-886f-e32da257adcc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-aab753-469nk\" (UID: \"b15f8554-6d3e-46bf-886f-e32da257adcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aab753-469nk"
Nov 03 18:14:07.356720 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:07.355808    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:14:07.356720 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:07.355909    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:14:07.356720 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:07.356033    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:14:07.356720 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:07.356122    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:14:07.523468 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:07.523421    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhvzz\" (UniqueName: \"kubernetes.io/projected/b15f8554-6d3e-46bf-886f-e32da257adcc-kube-api-access-qhvzz\") pod \"maas-api-revoke-keys-e2e-worker-w6-aab753-469nk\" (UID: \"b15f8554-6d3e-46bf-886f-e32da257adcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aab753-469nk"
Nov 03 18:14:07.559806 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:07.559734    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b15f8554-6d3e-46bf-886f-e32da257adcc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-aab753-469nk\" (UID: \"b15f8554-6d3e-46bf-886f-e32da257adcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aab753-469nk"
Nov 03 18:14:07.559991 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:07.559914    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b15f8554-6d3e-46bf-886f-e32da257adcc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-aab753-469nk\" (UID: \"b15f8554-6d3e-46bf-886f-e32da257adcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aab753-469nk"
Nov 03 18:14:07.568449 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:07.560988    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b15f8554-6d3e-46bf-886f-e32da257adcc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-aab753-469nk\" (UID: \"b15f8554-6d3e-46bf-886f-e32da257adcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aab753-469nk"
Nov 03 18:14:07.572033 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:07.571994    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b15f8554-6d3e-46bf-886f-e32da257adcc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-aab753-469nk\" (UID: \"b15f8554-6d3e-46bf-886f-e32da257adcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aab753-469nk"
Nov 03 18:14:07.625345 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:07.625276    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhvzz\" (UniqueName: \"kubernetes.io/projected/b15f8554-6d3e-46bf-886f-e32da257adcc-kube-api-access-qhvzz\") pod \"maas-api-revoke-keys-e2e-worker-w6-aab753-469nk\" (UID: \"b15f8554-6d3e-46bf-886f-e32da257adcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aab753-469nk"
Nov 03 18:14:07.705840 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:07.692864    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aab753-469nk"
Nov 03 18:14:08.495486 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:08.495442    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aab753-469nk"]
Nov 03 18:14:08.501712 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:14:08.501674    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb15f8554_6d3e_46bf_886f_e32da257adcc.slice/crio-e108828ba14f4e3f20244aeda418fad88905db4aec51b7b56febf90b40656e38 WatchSource:0}: Error finding container e108828ba14f4e3f20244aeda418fad88905db4aec51b7b56febf90b40656e38: Status 404 returned error can't find the container with id e108828ba14f4e3f20244aeda418fad88905db4aec51b7b56febf90b40656e38
Nov 03 18:14:08.808414 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:08.808387    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-0ccbb9-57dcc5ff7-zcv75"
Nov 03 18:14:08.895732 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:08.895700    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b86eb9eb-969a-4ac5-a947-0f1d32ca5b72-kube-api-access-4dln4\" (UniqueName: \"kubernetes.io/projected/b86eb9eb-969a-4ac5-a947-0f1d32ca5b72-kube-api-access-4dln4\") pod \"b86eb9eb-969a-4ac5-a947-0f1d32ca5b72\" (UID: \"b86eb9eb-969a-4ac5-a947-0f1d32ca5b72\") "
Nov 03 18:14:08.895905 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:08.895880    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b86eb9eb-969a-4ac5-a947-0f1d32ca5b72-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b86eb9eb-969a-4ac5-a947-0f1d32ca5b72-plugins-config-volume\") pod \"b86eb9eb-969a-4ac5-a947-0f1d32ca5b72\" (UID: \"b86eb9eb-969a-4ac5-a947-0f1d32ca5b72\") "
Nov 03 18:14:08.896842 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:08.896812    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b86eb9eb-969a-4ac5-a947-0f1d32ca5b72-plugins-config-volume" pod "b86eb9eb-969a-4ac5-a947-0f1d32ca5b72" (UID: "b86eb9eb-969a-4ac5-a947-0f1d32ca5b72"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:14:08.902020 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:08.901935    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b86eb9eb-969a-4ac5-a947-0f1d32ca5b72-kube-api-access-4dln4" pod "b86eb9eb-969a-4ac5-a947-0f1d32ca5b72" (UID: "b86eb9eb-969a-4ac5-a947-0f1d32ca5b72"). InnerVolumeSpecName "kube-api-access-4dln4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:08.916779 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:08.916738    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c1b2d2-6557dc56cf-kc2p2"
Nov 03 18:14:08.921171 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:08.921144    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c1b2d2-7c79954dcf-7htl8"
Nov 03 18:14:08.927418 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:08.927390    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0ccbb9-6b4b8ccc7f-q47rc"
Nov 03 18:14:08.997517 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:08.997417    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7580e5e-cc8b-412e-99f0-3a71c762e618-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7580e5e-cc8b-412e-99f0-3a71c762e618-plugins-config-volume\") pod \"e7580e5e-cc8b-412e-99f0-3a71c762e618\" (UID: \"e7580e5e-cc8b-412e-99f0-3a71c762e618\") "
Nov 03 18:14:08.997517 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:08.997474    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7580e5e-cc8b-412e-99f0-3a71c762e618-kube-api-access-thwfg\" (UniqueName: \"kubernetes.io/projected/e7580e5e-cc8b-412e-99f0-3a71c762e618-kube-api-access-thwfg\") pod \"e7580e5e-cc8b-412e-99f0-3a71c762e618\" (UID: \"e7580e5e-cc8b-412e-99f0-3a71c762e618\") "
Nov 03 18:14:08.997941 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:08.997913    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7580e5e-cc8b-412e-99f0-3a71c762e618-plugins-config-volume" pod "e7580e5e-cc8b-412e-99f0-3a71c762e618" (UID: "e7580e5e-cc8b-412e-99f0-3a71c762e618"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:14:08.998513 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:08.998235    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dln4\" (UniqueName: \"kubernetes.io/projected/b86eb9eb-969a-4ac5-a947-0f1d32ca5b72-kube-api-access-4dln4\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:08.998513 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:08.998261    5315 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7580e5e-cc8b-412e-99f0-3a71c762e618-plugins-config-volume\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:08.998513 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:08.998277    5315 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b86eb9eb-969a-4ac5-a947-0f1d32ca5b72-plugins-config-volume\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:09.000034 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:09.000003    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7580e5e-cc8b-412e-99f0-3a71c762e618-kube-api-access-thwfg" pod "e7580e5e-cc8b-412e-99f0-3a71c762e618" (UID: "e7580e5e-cc8b-412e-99f0-3a71c762e618"). InnerVolumeSpecName "kube-api-access-thwfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:09.099378 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:09.099341    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thwfg\" (UniqueName: \"kubernetes.io/projected/e7580e5e-cc8b-412e-99f0-3a71c762e618-kube-api-access-thwfg\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:09.262806 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:09.262694    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aab753-469nk" event={"ID":"b15f8554-6d3e-46bf-886f-e32da257adcc","Type":"ContainerStarted","Data":"f0ad76a5fbeb9bf99b9de0f7adc53fa8e035349b19109278680f5276d8014fe3"}
Nov 03 18:14:09.262806 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:09.262762    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aab753-469nk" event={"ID":"b15f8554-6d3e-46bf-886f-e32da257adcc","Type":"ContainerStarted","Data":"e108828ba14f4e3f20244aeda418fad88905db4aec51b7b56febf90b40656e38"}
Nov 03 18:14:09.264682 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:09.264651    5315 generic.go:356] "Generic (PLEG): container finished" podID="b86eb9eb-969a-4ac5-a947-0f1d32ca5b72" containerID="739bb1e5c2fba8967292d6a0db43d1eda8dfcd7af96cf7a2719ff6ca72202205" exitCode=0
Nov 03 18:14:09.264813 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:09.264701    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-0ccbb9-57dcc5ff7-zcv75"
Nov 03 18:14:09.264813 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:09.264734    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-0ccbb9-57dcc5ff7-zcv75" event={"ID":"b86eb9eb-969a-4ac5-a947-0f1d32ca5b72","Type":"ContainerDied","Data":"739bb1e5c2fba8967292d6a0db43d1eda8dfcd7af96cf7a2719ff6ca72202205"}
Nov 03 18:14:09.264813 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:09.264782    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-0ccbb9-57dcc5ff7-zcv75" event={"ID":"b86eb9eb-969a-4ac5-a947-0f1d32ca5b72","Type":"ContainerDied","Data":"16bbff5255624217845bcc0284d39dae1f8c28554a996962d852a1a1608b93da"}
Nov 03 18:14:09.264813 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:09.264810    5315 scope.go:122] "RemoveContainer" containerID="739bb1e5c2fba8967292d6a0db43d1eda8dfcd7af96cf7a2719ff6ca72202205"
Nov 03 18:14:09.266328 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:09.266300    5315 generic.go:356] "Generic (PLEG): container finished" podID="e7580e5e-cc8b-412e-99f0-3a71c762e618" containerID="4f586d9a86d82eca3d2d65bae3fae862f698adcdf5d24d6628fd2d81bbc45a69" exitCode=0
Nov 03 18:14:09.266440 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:09.266328    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0ccbb9-6b4b8ccc7f-q47rc" event={"ID":"e7580e5e-cc8b-412e-99f0-3a71c762e618","Type":"ContainerDied","Data":"4f586d9a86d82eca3d2d65bae3fae862f698adcdf5d24d6628fd2d81bbc45a69"}
Nov 03 18:14:09.266440 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:09.266357    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0ccbb9-6b4b8ccc7f-q47rc"
Nov 03 18:14:09.266440 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:09.266378    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0ccbb9-6b4b8ccc7f-q47rc" event={"ID":"e7580e5e-cc8b-412e-99f0-3a71c762e618","Type":"ContainerDied","Data":"15e6d9e21da3a3bbb398139f46f0d8827b4a38ed4703ed0ef5df8c1e347cf7c7"}
Nov 03 18:14:09.279890 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:09.279862    5315 scope.go:122] "RemoveContainer" containerID="739bb1e5c2fba8967292d6a0db43d1eda8dfcd7af96cf7a2719ff6ca72202205"
Nov 03 18:14:09.281098 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:14:09.281067    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"739bb1e5c2fba8967292d6a0db43d1eda8dfcd7af96cf7a2719ff6ca72202205\": container with ID starting with 739bb1e5c2fba8967292d6a0db43d1eda8dfcd7af96cf7a2719ff6ca72202205 not found: ID does not exist" containerID="739bb1e5c2fba8967292d6a0db43d1eda8dfcd7af96cf7a2719ff6ca72202205"
Nov 03 18:14:09.281229 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:09.281108    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"739bb1e5c2fba8967292d6a0db43d1eda8dfcd7af96cf7a2719ff6ca72202205"} err="failed to get container status \"739bb1e5c2fba8967292d6a0db43d1eda8dfcd7af96cf7a2719ff6ca72202205\": rpc error: code = NotFound desc = could not find container \"739bb1e5c2fba8967292d6a0db43d1eda8dfcd7af96cf7a2719ff6ca72202205\": container with ID starting with 739bb1e5c2fba8967292d6a0db43d1eda8dfcd7af96cf7a2719ff6ca72202205 not found: ID does not exist"
Nov 03 18:14:09.281229 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:09.281134    5315 scope.go:122] "RemoveContainer" containerID="4f586d9a86d82eca3d2d65bae3fae862f698adcdf5d24d6628fd2d81bbc45a69"
Nov 03 18:14:09.293099 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:09.293011    5315 scope.go:122] "RemoveContainer" containerID="4f586d9a86d82eca3d2d65bae3fae862f698adcdf5d24d6628fd2d81bbc45a69"
Nov 03 18:14:09.293470 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:14:09.293416    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f586d9a86d82eca3d2d65bae3fae862f698adcdf5d24d6628fd2d81bbc45a69\": container with ID starting with 4f586d9a86d82eca3d2d65bae3fae862f698adcdf5d24d6628fd2d81bbc45a69 not found: ID does not exist" containerID="4f586d9a86d82eca3d2d65bae3fae862f698adcdf5d24d6628fd2d81bbc45a69"
Nov 03 18:14:09.293623 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:09.293484    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f586d9a86d82eca3d2d65bae3fae862f698adcdf5d24d6628fd2d81bbc45a69"} err="failed to get container status \"4f586d9a86d82eca3d2d65bae3fae862f698adcdf5d24d6628fd2d81bbc45a69\": rpc error: code = NotFound desc = could not find container \"4f586d9a86d82eca3d2d65bae3fae862f698adcdf5d24d6628fd2d81bbc45a69\": container with ID starting with 4f586d9a86d82eca3d2d65bae3fae862f698adcdf5d24d6628fd2d81bbc45a69 not found: ID does not exist"
Nov 03 18:14:09.371147 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:09.371092    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aab753-469nk" podStartSLOduration=2.371076026 podStartE2EDuration="2.371076026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:14:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:14:09.369539161 +0000 UTC m=+1688.385737232" watchObservedRunningTime="2026-11-03 18:14:09.371076026 +0000 UTC m=+1688.387274095"
Nov 03 18:14:09.475253 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:09.475208    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-0ccbb9-6b4b8ccc7f-q47rc"]
Nov 03 18:14:09.481320 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:09.481295    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-0ccbb9-6b4b8ccc7f-q47rc"]
Nov 03 18:14:09.581573 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:09.581541    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-0ccbb9-57dcc5ff7-zcv75"]
Nov 03 18:14:09.586275 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:09.586246    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-0ccbb9-57dcc5ff7-zcv75"]
Nov 03 18:14:10.275959 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:10.275927    5315 generic.go:356] "Generic (PLEG): container finished" podID="b15f8554-6d3e-46bf-886f-e32da257adcc" containerID="f0ad76a5fbeb9bf99b9de0f7adc53fa8e035349b19109278680f5276d8014fe3" exitCode=0
Nov 03 18:14:10.276269 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:10.276015    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aab753-469nk" event={"ID":"b15f8554-6d3e-46bf-886f-e32da257adcc","Type":"ContainerDied","Data":"f0ad76a5fbeb9bf99b9de0f7adc53fa8e035349b19109278680f5276d8014fe3"}
Nov 03 18:14:11.107301 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:11.107255    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b86eb9eb-969a-4ac5-a947-0f1d32ca5b72" path="/var/lib/kubelet/pods/b86eb9eb-969a-4ac5-a947-0f1d32ca5b72/volumes"
Nov 03 18:14:11.107985 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:11.107963    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7580e5e-cc8b-412e-99f0-3a71c762e618" path="/var/lib/kubelet/pods/e7580e5e-cc8b-412e-99f0-3a71c762e618/volumes"
Nov 03 18:14:11.586647 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:11.579315    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aab753-469nk"
Nov 03 18:14:11.680420 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:11.680385    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv"]
Nov 03 18:14:11.689216 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:11.689185    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7"]
Nov 03 18:14:11.698302 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:11.698268    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b15f8554-6d3e-46bf-886f-e32da257adcc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b15f8554-6d3e-46bf-886f-e32da257adcc-maas-api-xks-ca\") pod \"b15f8554-6d3e-46bf-886f-e32da257adcc\" (UID: \"b15f8554-6d3e-46bf-886f-e32da257adcc\") "
Nov 03 18:14:11.698481 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:11.698433    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b15f8554-6d3e-46bf-886f-e32da257adcc-kube-api-access-qhvzz\" (UniqueName: \"kubernetes.io/projected/b15f8554-6d3e-46bf-886f-e32da257adcc-kube-api-access-qhvzz\") pod \"b15f8554-6d3e-46bf-886f-e32da257adcc\" (UID: \"b15f8554-6d3e-46bf-886f-e32da257adcc\") "
Nov 03 18:14:11.701661 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:11.698549    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b15f8554-6d3e-46bf-886f-e32da257adcc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b15f8554-6d3e-46bf-886f-e32da257adcc-maas-api-openshift-ca\") pod \"b15f8554-6d3e-46bf-886f-e32da257adcc\" (UID: \"b15f8554-6d3e-46bf-886f-e32da257adcc\") "
Nov 03 18:14:11.701661 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:11.698932    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b15f8554-6d3e-46bf-886f-e32da257adcc-maas-api-openshift-ca" pod "b15f8554-6d3e-46bf-886f-e32da257adcc" (UID: "b15f8554-6d3e-46bf-886f-e32da257adcc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:14:11.701661 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:11.699294    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b15f8554-6d3e-46bf-886f-e32da257adcc-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:11.701661 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:11.700779    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b15f8554-6d3e-46bf-886f-e32da257adcc-maas-api-xks-ca" pod "b15f8554-6d3e-46bf-886f-e32da257adcc" (UID: "b15f8554-6d3e-46bf-886f-e32da257adcc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:14:11.701661 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:11.700931    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b15f8554-6d3e-46bf-886f-e32da257adcc-kube-api-access-qhvzz" pod "b15f8554-6d3e-46bf-886f-e32da257adcc" (UID: "b15f8554-6d3e-46bf-886f-e32da257adcc"). InnerVolumeSpecName "kube-api-access-qhvzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:11.800958 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:11.800919    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhvzz\" (UniqueName: \"kubernetes.io/projected/b15f8554-6d3e-46bf-886f-e32da257adcc-kube-api-access-qhvzz\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:11.800958 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:11.800956    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b15f8554-6d3e-46bf-886f-e32da257adcc-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:12.294078 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:12.294049    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aab753-469nk"
Nov 03 18:14:12.294480 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:12.294049    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aab753-469nk" event={"ID":"b15f8554-6d3e-46bf-886f-e32da257adcc","Type":"ContainerDied","Data":"e108828ba14f4e3f20244aeda418fad88905db4aec51b7b56febf90b40656e38"}
Nov 03 18:14:12.294480 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:12.294175    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e108828ba14f4e3f20244aeda418fad88905db4aec51b7b56febf90b40656e38"
Nov 03 18:14:13.476970 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:13.476935    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-260dd0-76766ff95-8999v"]
Nov 03 18:14:13.521624 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:13.521583    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g"]
Nov 03 18:14:13.522241 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:13.522219    5315 cpu_manager.go:414] "containerMap: removing container" podUID="e7580e5e-cc8b-412e-99f0-3a71c762e618" containerName="payload-pre-processing"
Nov 03 18:14:13.522296 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:13.522237    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7580e5e-cc8b-412e-99f0-3a71c762e618" containerName="payload-pre-processing"
Nov 03 18:14:13.522296 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:13.522288    5315 cpu_manager.go:414] "containerMap: removing container" podUID="b15f8554-6d3e-46bf-886f-e32da257adcc" containerName="revoke-keys"
Nov 03 18:14:13.522361 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:13.522296    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b15f8554-6d3e-46bf-886f-e32da257adcc" containerName="revoke-keys"
Nov 03 18:14:13.522361 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:13.522313    5315 cpu_manager.go:414] "containerMap: removing container" podUID="d7899300-736f-4a03-aef7-253d7a1b1e2f" containerName="istio-proxy"
Nov 03 18:14:13.522361 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:13.522323    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7899300-736f-4a03-aef7-253d7a1b1e2f" containerName="istio-proxy"
Nov 03 18:14:13.522361 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:13.522330    5315 cpu_manager.go:414] "containerMap: removing container" podUID="b86eb9eb-969a-4ac5-a947-0f1d32ca5b72" containerName="payload-processing"
Nov 03 18:14:13.522361 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:13.522335    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b86eb9eb-969a-4ac5-a947-0f1d32ca5b72" containerName="payload-processing"
Nov 03 18:14:13.522516 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:13.522432    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="b15f8554-6d3e-46bf-886f-e32da257adcc" containerName="revoke-keys"
Nov 03 18:14:13.522516 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:13.522440    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="b86eb9eb-969a-4ac5-a947-0f1d32ca5b72" containerName="payload-processing"
Nov 03 18:14:13.522516 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:13.522447    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7580e5e-cc8b-412e-99f0-3a71c762e618" containerName="payload-pre-processing"
Nov 03 18:14:13.525830 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:13.525052    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g"
Nov 03 18:14:13.573910 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:13.573875    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-232f9e-openshift-default-dockercfg-657fr\""
Nov 03 18:14:13.574844 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:13.574815    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-260dd0-5469bdccd7-82scv"]
Nov 03 18:14:13.592118 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:13.592066    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g"]
Nov 03 18:14:13.603278 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:13.603244    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/915e71a9-f7c9-4be4-929a-af6be579b6bd-istiod-ca-cert\") pod \"e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g\" (UID: \"915e71a9-f7c9-4be4-929a-af6be579b6bd\") " pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g"
Nov 03 18:14:13.603446 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:13.603292    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/915e71a9-f7c9-4be4-929a-af6be579b6bd-istio-podinfo\") pod \"e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g\" (UID: \"915e71a9-f7c9-4be4-929a-af6be579b6bd\") " pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g"
Nov 03 18:14:13.603446 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:13.603418    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/915e71a9-f7c9-4be4-929a-af6be579b6bd-credential-socket\") pod \"e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g\" (UID: \"915e71a9-f7c9-4be4-929a-af6be579b6bd\") " pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g"
Nov 03 18:14:13.603577 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:13.603490    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/915e71a9-f7c9-4be4-929a-af6be579b6bd-workload-socket\") pod \"e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g\" (UID: \"915e71a9-f7c9-4be4-929a-af6be579b6bd\") " pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g"
Nov 03 18:14:13.603577 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:13.603569    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/915e71a9-f7c9-4be4-929a-af6be579b6bd-workload-certs\") pod \"e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g\" (UID: \"915e71a9-f7c9-4be4-929a-af6be579b6bd\") " pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g"
Nov 03 18:14:13.603772 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:13.603740    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/915e71a9-f7c9-4be4-929a-af6be579b6bd-istio-data\") pod \"e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g\" (UID: \"915e71a9-f7c9-4be4-929a-af6be579b6bd\") " pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g"
Nov 03 18:14:13.603847 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:13.603804    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/915e71a9-f7c9-4be4-929a-af6be579b6bd-istio-token\") pod \"e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g\" (UID: \"915e71a9-f7c9-4be4-929a-af6be579b6bd\") " pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g"
Nov 03 18:14:13.603901 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:13.603880    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/915e71a9-f7c9-4be4-929a-af6be579b6bd-istio-envoy\") pod \"e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g\" (UID: \"915e71a9-f7c9-4be4-929a-af6be579b6bd\") " pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g"
Nov 03 18:14:13.603957 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:13.603909    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zxcd\" (UniqueName: \"kubernetes.io/projected/915e71a9-f7c9-4be4-929a-af6be579b6bd-kube-api-access-7zxcd\") pod \"e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g\" (UID: \"915e71a9-f7c9-4be4-929a-af6be579b6bd\") " pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g"
Nov 03 18:14:13.705419 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:13.705380    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/915e71a9-f7c9-4be4-929a-af6be579b6bd-credential-socket\") pod \"e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g\" (UID: \"915e71a9-f7c9-4be4-929a-af6be579b6bd\") " pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g"
Nov 03 18:14:13.705638 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:13.705437    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/915e71a9-f7c9-4be4-929a-af6be579b6bd-workload-socket\") pod \"e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g\" (UID: \"915e71a9-f7c9-4be4-929a-af6be579b6bd\") " pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g"
Nov 03 18:14:13.705638 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:13.705498    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/915e71a9-f7c9-4be4-929a-af6be579b6bd-workload-certs\") pod \"e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g\" (UID: \"915e71a9-f7c9-4be4-929a-af6be579b6bd\") " pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g"
Nov 03 18:14:13.705638 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:13.705573    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/915e71a9-f7c9-4be4-929a-af6be579b6bd-istio-data\") pod \"e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g\" (UID: \"915e71a9-f7c9-4be4-929a-af6be579b6bd\") " pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g"
Nov 03 18:14:13.705638 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:13.705633    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/915e71a9-f7c9-4be4-929a-af6be579b6bd-istio-token\") pod \"e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g\" (UID: \"915e71a9-f7c9-4be4-929a-af6be579b6bd\") " pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g"
Nov 03 18:14:13.705855 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:13.705695    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/915e71a9-f7c9-4be4-929a-af6be579b6bd-istio-envoy\") pod \"e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g\" (UID: \"915e71a9-f7c9-4be4-929a-af6be579b6bd\") " pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g"
Nov 03 18:14:13.705855 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:13.705726    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zxcd\" (UniqueName: \"kubernetes.io/projected/915e71a9-f7c9-4be4-929a-af6be579b6bd-kube-api-access-7zxcd\") pod \"e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g\" (UID: \"915e71a9-f7c9-4be4-929a-af6be579b6bd\") " pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g"
Nov 03 18:14:13.705855 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:13.705812    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/915e71a9-f7c9-4be4-929a-af6be579b6bd-istiod-ca-cert\") pod \"e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g\" (UID: \"915e71a9-f7c9-4be4-929a-af6be579b6bd\") " pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g"
Nov 03 18:14:13.706018 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:13.705864    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/915e71a9-f7c9-4be4-929a-af6be579b6bd-istio-podinfo\") pod \"e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g\" (UID: \"915e71a9-f7c9-4be4-929a-af6be579b6bd\") " pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g"
Nov 03 18:14:13.706018 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:13.705860    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/915e71a9-f7c9-4be4-929a-af6be579b6bd-workload-socket\") pod \"e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g\" (UID: \"915e71a9-f7c9-4be4-929a-af6be579b6bd\") " pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g"
Nov 03 18:14:13.706267 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:13.706220    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/915e71a9-f7c9-4be4-929a-af6be579b6bd-istio-data\") pod \"e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g\" (UID: \"915e71a9-f7c9-4be4-929a-af6be579b6bd\") " pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g"
Nov 03 18:14:13.706267 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:13.706241    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/915e71a9-f7c9-4be4-929a-af6be579b6bd-credential-socket\") pod \"e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g\" (UID: \"915e71a9-f7c9-4be4-929a-af6be579b6bd\") " pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g"
Nov 03 18:14:13.706462 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:13.706420    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/915e71a9-f7c9-4be4-929a-af6be579b6bd-workload-certs\") pod \"e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g\" (UID: \"915e71a9-f7c9-4be4-929a-af6be579b6bd\") " pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g"
Nov 03 18:14:13.706746 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:13.706726    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/915e71a9-f7c9-4be4-929a-af6be579b6bd-istiod-ca-cert\") pod \"e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g\" (UID: \"915e71a9-f7c9-4be4-929a-af6be579b6bd\") " pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g"
Nov 03 18:14:13.708552 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:13.708529    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/915e71a9-f7c9-4be4-929a-af6be579b6bd-istio-envoy\") pod \"e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g\" (UID: \"915e71a9-f7c9-4be4-929a-af6be579b6bd\") " pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g"
Nov 03 18:14:13.708750 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:13.708733    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/915e71a9-f7c9-4be4-929a-af6be579b6bd-istio-podinfo\") pod \"e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g\" (UID: \"915e71a9-f7c9-4be4-929a-af6be579b6bd\") " pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g"
Nov 03 18:14:13.805346 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:13.805270    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/915e71a9-f7c9-4be4-929a-af6be579b6bd-istio-token\") pod \"e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g\" (UID: \"915e71a9-f7c9-4be4-929a-af6be579b6bd\") " pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g"
Nov 03 18:14:13.851645 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:13.851114    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zxcd\" (UniqueName: \"kubernetes.io/projected/915e71a9-f7c9-4be4-929a-af6be579b6bd-kube-api-access-7zxcd\") pod \"e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g\" (UID: \"915e71a9-f7c9-4be4-929a-af6be579b6bd\") " pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g"
Nov 03 18:14:14.163684 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:14.163206    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g"
Nov 03 18:14:14.647210 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:14.647164    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g"]
Nov 03 18:14:14.652654 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:14:14.652588    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod915e71a9_f7c9_4be4_929a_af6be579b6bd.slice/crio-7e936087cb388bc5625bd53ab6d23b4a86ab6db0da6974a7015d62aeed2e87ff WatchSource:0}: Error finding container 7e936087cb388bc5625bd53ab6d23b4a86ab6db0da6974a7015d62aeed2e87ff: Status 404 returned error can't find the container with id 7e936087cb388bc5625bd53ab6d23b4a86ab6db0da6974a7015d62aeed2e87ff
Nov 03 18:14:14.657425 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:14.657388    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:14:14.657533 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:14.657461    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:14:14.657533 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:14.657491    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:14:15.367245 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:15.367202    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g" event={"ID":"915e71a9-f7c9-4be4-929a-af6be579b6bd","Type":"ContainerStarted","Data":"8075f30f685a50373bf0aad4c0e226ba8e9fae327819f934e88dc058ed053ce1"}
Nov 03 18:14:15.367550 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:15.367526    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g" event={"ID":"915e71a9-f7c9-4be4-929a-af6be579b6bd","Type":"ContainerStarted","Data":"7e936087cb388bc5625bd53ab6d23b4a86ab6db0da6974a7015d62aeed2e87ff"}
Nov 03 18:14:15.515634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:15.513972    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g" podStartSLOduration=2.513944142 podStartE2EDuration="2.513944142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:14:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:14:15.509094711 +0000 UTC m=+1694.525292792" watchObservedRunningTime="2026-11-03 18:14:15.513944142 +0000 UTC m=+1694.530142213"
Nov 03 18:14:16.119338 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.119297    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-0adcbc-695b8c899-b9rjk"]
Nov 03 18:14:16.163836 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.163795    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g"
Nov 03 18:14:16.169827 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.169793    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g"
Nov 03 18:14:16.230828 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.230785    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-0adcbc-b4854959-pwnvs"]
Nov 03 18:14:16.380224 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.380117    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g"
Nov 03 18:14:16.381242 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.381220    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g"
Nov 03 18:14:16.393857 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.393815    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7" podUID="09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb" containerName="main" containerID="cri-o://83357fb07c4a01250285ec6a651a109a9f966ad2f35ef294f0cab0100e8cd9ea" gracePeriod=45
Nov 03 18:14:16.530061 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.530020    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv"]
Nov 03 18:14:16.552686 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.547444    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv"
Nov 03 18:14:16.561502 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.557629    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl"]
Nov 03 18:14:16.588633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.585320    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-istiod-ca-cert\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv\" (UID: \"a36c9b31-8ab0-449f-8fb4-87a702ab7abf\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv"
Nov 03 18:14:16.588633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.585400    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-istio-podinfo\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv\" (UID: \"a36c9b31-8ab0-449f-8fb4-87a702ab7abf\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv"
Nov 03 18:14:16.588633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.585459    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-istio-envoy\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv\" (UID: \"a36c9b31-8ab0-449f-8fb4-87a702ab7abf\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv"
Nov 03 18:14:16.588633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.585511    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-istio-token\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv\" (UID: \"a36c9b31-8ab0-449f-8fb4-87a702ab7abf\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv"
Nov 03 18:14:16.588633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.585581    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-workload-certs\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv\" (UID: \"a36c9b31-8ab0-449f-8fb4-87a702ab7abf\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv"
Nov 03 18:14:16.588633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.585777    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-workload-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv\" (UID: \"a36c9b31-8ab0-449f-8fb4-87a702ab7abf\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv"
Nov 03 18:14:16.588633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.585833    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ptf5\" (UniqueName: \"kubernetes.io/projected/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-kube-api-access-2ptf5\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv\" (UID: \"a36c9b31-8ab0-449f-8fb4-87a702ab7abf\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv"
Nov 03 18:14:16.588633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.585865    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-credential-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv\" (UID: \"a36c9b31-8ab0-449f-8fb4-87a702ab7abf\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv"
Nov 03 18:14:16.588633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.585923    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-istio-data\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv\" (UID: \"a36c9b31-8ab0-449f-8fb4-87a702ab7abf\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv"
Nov 03 18:14:16.604377 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.599001    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-0ccbb9-openshift-default-dockercfg-8dgdl\""
Nov 03 18:14:16.688529 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.687704    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-istio-token\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv\" (UID: \"a36c9b31-8ab0-449f-8fb4-87a702ab7abf\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv"
Nov 03 18:14:16.688947 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.688875    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-workload-certs\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv\" (UID: \"a36c9b31-8ab0-449f-8fb4-87a702ab7abf\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv"
Nov 03 18:14:16.689956 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.689918    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-workload-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv\" (UID: \"a36c9b31-8ab0-449f-8fb4-87a702ab7abf\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv"
Nov 03 18:14:16.690162 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.690140    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ptf5\" (UniqueName: \"kubernetes.io/projected/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-kube-api-access-2ptf5\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv\" (UID: \"a36c9b31-8ab0-449f-8fb4-87a702ab7abf\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv"
Nov 03 18:14:16.691791 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.690284    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-credential-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv\" (UID: \"a36c9b31-8ab0-449f-8fb4-87a702ab7abf\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv"
Nov 03 18:14:16.691992 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.691968    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-istio-data\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv\" (UID: \"a36c9b31-8ab0-449f-8fb4-87a702ab7abf\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv"
Nov 03 18:14:16.693389 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.693369    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-istiod-ca-cert\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv\" (UID: \"a36c9b31-8ab0-449f-8fb4-87a702ab7abf\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv"
Nov 03 18:14:16.693578 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.693563    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-istio-podinfo\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv\" (UID: \"a36c9b31-8ab0-449f-8fb4-87a702ab7abf\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv"
Nov 03 18:14:16.693735 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.693717    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-istio-envoy\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv\" (UID: \"a36c9b31-8ab0-449f-8fb4-87a702ab7abf\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv"
Nov 03 18:14:16.693953 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.692959    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-credential-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv\" (UID: \"a36c9b31-8ab0-449f-8fb4-87a702ab7abf\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv"
Nov 03 18:14:16.694090 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.692974    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-workload-certs\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv\" (UID: \"a36c9b31-8ab0-449f-8fb4-87a702ab7abf\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv"
Nov 03 18:14:16.694188 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.693219    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-istio-data\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv\" (UID: \"a36c9b31-8ab0-449f-8fb4-87a702ab7abf\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv"
Nov 03 18:14:16.694296 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.692257    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb"]
Nov 03 18:14:16.695162 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.695114    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-istiod-ca-cert\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv\" (UID: \"a36c9b31-8ab0-449f-8fb4-87a702ab7abf\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv"
Nov 03 18:14:16.695288 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.692682    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-workload-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv\" (UID: \"a36c9b31-8ab0-449f-8fb4-87a702ab7abf\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv"
Nov 03 18:14:16.716298 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.716262    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-istio-podinfo\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv\" (UID: \"a36c9b31-8ab0-449f-8fb4-87a702ab7abf\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv"
Nov 03 18:14:16.716527 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.692352    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl"
Nov 03 18:14:16.738750 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.738715    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv"]
Nov 03 18:14:16.738997 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.738979    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb"]
Nov 03 18:14:16.739209 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.739191    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb"
Nov 03 18:14:16.770467 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.758727    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-istio-envoy\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv\" (UID: \"a36c9b31-8ab0-449f-8fb4-87a702ab7abf\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv"
Nov 03 18:14:16.809446 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.801254    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl"]
Nov 03 18:14:16.827649 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.824574    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-istio-token\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv\" (UID: \"a36c9b31-8ab0-449f-8fb4-87a702ab7abf\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv"
Nov 03 18:14:16.827649 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.827361    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ptf5\" (UniqueName: \"kubernetes.io/projected/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-kube-api-access-2ptf5\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv\" (UID: \"a36c9b31-8ab0-449f-8fb4-87a702ab7abf\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv"
Nov 03 18:14:16.867161 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.858849    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv"
Nov 03 18:14:16.904655 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.904546    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56w4t\" (UniqueName: \"kubernetes.io/projected/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-kube-api-access-56w4t\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb\" (UID: \"c4428ff6-1929-4fb0-82ee-ab6a785ca9e5\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb"
Nov 03 18:14:16.906798 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.905558    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-workload-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb\" (UID: \"c4428ff6-1929-4fb0-82ee-ab6a785ca9e5\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb"
Nov 03 18:14:16.906798 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.905597    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-workload-certs\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb\" (UID: \"c4428ff6-1929-4fb0-82ee-ab6a785ca9e5\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb"
Nov 03 18:14:16.906798 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.905674    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ede64d2a-75d4-4d21-8592-1680fb138e4e-istio-podinfo\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl\" (UID: \"ede64d2a-75d4-4d21-8592-1680fb138e4e\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl"
Nov 03 18:14:16.906798 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.905750    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-istiod-ca-cert\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb\" (UID: \"c4428ff6-1929-4fb0-82ee-ab6a785ca9e5\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb"
Nov 03 18:14:16.906798 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.905794    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ede64d2a-75d4-4d21-8592-1680fb138e4e-workload-certs\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl\" (UID: \"ede64d2a-75d4-4d21-8592-1680fb138e4e\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl"
Nov 03 18:14:16.906798 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.905951    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede64d2a-75d4-4d21-8592-1680fb138e4e-workload-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl\" (UID: \"ede64d2a-75d4-4d21-8592-1680fb138e4e\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl"
Nov 03 18:14:16.906798 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.906034    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ede64d2a-75d4-4d21-8592-1680fb138e4e-istio-data\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl\" (UID: \"ede64d2a-75d4-4d21-8592-1680fb138e4e\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl"
Nov 03 18:14:16.906798 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.906065    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-istio-data\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb\" (UID: \"c4428ff6-1929-4fb0-82ee-ab6a785ca9e5\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb"
Nov 03 18:14:16.906798 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.906097    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ede64d2a-75d4-4d21-8592-1680fb138e4e-istio-envoy\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl\" (UID: \"ede64d2a-75d4-4d21-8592-1680fb138e4e\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl"
Nov 03 18:14:16.906798 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.906123    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ede64d2a-75d4-4d21-8592-1680fb138e4e-istio-token\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl\" (UID: \"ede64d2a-75d4-4d21-8592-1680fb138e4e\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl"
Nov 03 18:14:16.906798 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.906195    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ede64d2a-75d4-4d21-8592-1680fb138e4e-istiod-ca-cert\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl\" (UID: \"ede64d2a-75d4-4d21-8592-1680fb138e4e\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl"
Nov 03 18:14:16.906798 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.906238    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede64d2a-75d4-4d21-8592-1680fb138e4e-credential-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl\" (UID: \"ede64d2a-75d4-4d21-8592-1680fb138e4e\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl"
Nov 03 18:14:16.906798 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.906304    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-istio-podinfo\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb\" (UID: \"c4428ff6-1929-4fb0-82ee-ab6a785ca9e5\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb"
Nov 03 18:14:16.906798 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.906341    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-credential-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb\" (UID: \"c4428ff6-1929-4fb0-82ee-ab6a785ca9e5\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb"
Nov 03 18:14:16.906798 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.906369    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-istio-envoy\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb\" (UID: \"c4428ff6-1929-4fb0-82ee-ab6a785ca9e5\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb"
Nov 03 18:14:16.906798 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.906483    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s44q\" (UniqueName: \"kubernetes.io/projected/ede64d2a-75d4-4d21-8592-1680fb138e4e-kube-api-access-2s44q\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl\" (UID: \"ede64d2a-75d4-4d21-8592-1680fb138e4e\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl"
Nov 03 18:14:16.906798 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.906536    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-istio-token\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb\" (UID: \"c4428ff6-1929-4fb0-82ee-ab6a785ca9e5\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb"
Nov 03 18:14:16.927086 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:16.926854    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7"
Nov 03 18:14:17.008258 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.008218    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s44q\" (UniqueName: \"kubernetes.io/projected/ede64d2a-75d4-4d21-8592-1680fb138e4e-kube-api-access-2s44q\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl\" (UID: \"ede64d2a-75d4-4d21-8592-1680fb138e4e\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl"
Nov 03 18:14:17.008445 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.008293    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-istio-token\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb\" (UID: \"c4428ff6-1929-4fb0-82ee-ab6a785ca9e5\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb"
Nov 03 18:14:17.008445 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.008428    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56w4t\" (UniqueName: \"kubernetes.io/projected/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-kube-api-access-56w4t\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb\" (UID: \"c4428ff6-1929-4fb0-82ee-ab6a785ca9e5\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb"
Nov 03 18:14:17.008627 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.008525    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-workload-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb\" (UID: \"c4428ff6-1929-4fb0-82ee-ab6a785ca9e5\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb"
Nov 03 18:14:17.008627 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.008559    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-workload-certs\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb\" (UID: \"c4428ff6-1929-4fb0-82ee-ab6a785ca9e5\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb"
Nov 03 18:14:17.008750 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.008651    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ede64d2a-75d4-4d21-8592-1680fb138e4e-istio-podinfo\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl\" (UID: \"ede64d2a-75d4-4d21-8592-1680fb138e4e\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl"
Nov 03 18:14:17.008805 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.008761    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-istiod-ca-cert\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb\" (UID: \"c4428ff6-1929-4fb0-82ee-ab6a785ca9e5\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb"
Nov 03 18:14:17.008856 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.008805    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ede64d2a-75d4-4d21-8592-1680fb138e4e-workload-certs\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl\" (UID: \"ede64d2a-75d4-4d21-8592-1680fb138e4e\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl"
Nov 03 18:14:17.008926 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.008901    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede64d2a-75d4-4d21-8592-1680fb138e4e-workload-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl\" (UID: \"ede64d2a-75d4-4d21-8592-1680fb138e4e\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl"
Nov 03 18:14:17.008983 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.008957    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ede64d2a-75d4-4d21-8592-1680fb138e4e-istio-data\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl\" (UID: \"ede64d2a-75d4-4d21-8592-1680fb138e4e\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl"
Nov 03 18:14:17.009037 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.008988    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-istio-data\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb\" (UID: \"c4428ff6-1929-4fb0-82ee-ab6a785ca9e5\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb"
Nov 03 18:14:17.009037 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.009021    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ede64d2a-75d4-4d21-8592-1680fb138e4e-istio-envoy\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl\" (UID: \"ede64d2a-75d4-4d21-8592-1680fb138e4e\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl"
Nov 03 18:14:17.009138 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.009046    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ede64d2a-75d4-4d21-8592-1680fb138e4e-istio-token\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl\" (UID: \"ede64d2a-75d4-4d21-8592-1680fb138e4e\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl"
Nov 03 18:14:17.009138 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.009119    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ede64d2a-75d4-4d21-8592-1680fb138e4e-istiod-ca-cert\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl\" (UID: \"ede64d2a-75d4-4d21-8592-1680fb138e4e\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl"
Nov 03 18:14:17.009254 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.009162    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede64d2a-75d4-4d21-8592-1680fb138e4e-credential-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl\" (UID: \"ede64d2a-75d4-4d21-8592-1680fb138e4e\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl"
Nov 03 18:14:17.009254 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.009230    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-istio-podinfo\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb\" (UID: \"c4428ff6-1929-4fb0-82ee-ab6a785ca9e5\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb"
Nov 03 18:14:17.009356 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.009299    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-credential-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb\" (UID: \"c4428ff6-1929-4fb0-82ee-ab6a785ca9e5\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb"
Nov 03 18:14:17.009356 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.009333    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-istio-envoy\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb\" (UID: \"c4428ff6-1929-4fb0-82ee-ab6a785ca9e5\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb"
Nov 03 18:14:17.009874 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.009849    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-workload-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb\" (UID: \"c4428ff6-1929-4fb0-82ee-ab6a785ca9e5\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb"
Nov 03 18:14:17.010184 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.010168    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-istio-data\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb\" (UID: \"c4428ff6-1929-4fb0-82ee-ab6a785ca9e5\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb"
Nov 03 18:14:17.010302 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.010277    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-istiod-ca-cert\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb\" (UID: \"c4428ff6-1929-4fb0-82ee-ab6a785ca9e5\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb"
Nov 03 18:14:17.011427 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.011336    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ede64d2a-75d4-4d21-8592-1680fb138e4e-workload-certs\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl\" (UID: \"ede64d2a-75d4-4d21-8592-1680fb138e4e\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl"
Nov 03 18:14:17.020656 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.012460    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ede64d2a-75d4-4d21-8592-1680fb138e4e-istio-data\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl\" (UID: \"ede64d2a-75d4-4d21-8592-1680fb138e4e\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl"
Nov 03 18:14:17.020656 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.012573    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-workload-certs\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb\" (UID: \"c4428ff6-1929-4fb0-82ee-ab6a785ca9e5\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb"
Nov 03 18:14:17.020656 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.013876    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-istio-envoy\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb\" (UID: \"c4428ff6-1929-4fb0-82ee-ab6a785ca9e5\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb"
Nov 03 18:14:17.020656 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.014379    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ede64d2a-75d4-4d21-8592-1680fb138e4e-istio-podinfo\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl\" (UID: \"ede64d2a-75d4-4d21-8592-1680fb138e4e\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl"
Nov 03 18:14:17.020656 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.014888    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede64d2a-75d4-4d21-8592-1680fb138e4e-workload-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl\" (UID: \"ede64d2a-75d4-4d21-8592-1680fb138e4e\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl"
Nov 03 18:14:17.020656 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.015197    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede64d2a-75d4-4d21-8592-1680fb138e4e-credential-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl\" (UID: \"ede64d2a-75d4-4d21-8592-1680fb138e4e\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl"
Nov 03 18:14:17.020656 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.015442    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ede64d2a-75d4-4d21-8592-1680fb138e4e-istiod-ca-cert\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl\" (UID: \"ede64d2a-75d4-4d21-8592-1680fb138e4e\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl"
Nov 03 18:14:17.020656 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.015746    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-credential-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb\" (UID: \"c4428ff6-1929-4fb0-82ee-ab6a785ca9e5\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb"
Nov 03 18:14:17.031942 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.031903    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-istio-podinfo\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb\" (UID: \"c4428ff6-1929-4fb0-82ee-ab6a785ca9e5\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb"
Nov 03 18:14:17.032199 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.032170    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ede64d2a-75d4-4d21-8592-1680fb138e4e-istio-envoy\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl\" (UID: \"ede64d2a-75d4-4d21-8592-1680fb138e4e\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl"
Nov 03 18:14:17.065516 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.065483    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s44q\" (UniqueName: \"kubernetes.io/projected/ede64d2a-75d4-4d21-8592-1680fb138e4e-kube-api-access-2s44q\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl\" (UID: \"ede64d2a-75d4-4d21-8592-1680fb138e4e\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl"
Nov 03 18:14:17.066644 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.066600    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56w4t\" (UniqueName: \"kubernetes.io/projected/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-kube-api-access-56w4t\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb\" (UID: \"c4428ff6-1929-4fb0-82ee-ab6a785ca9e5\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb"
Nov 03 18:14:17.067033 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.067010    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-istio-token\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb\" (UID: \"c4428ff6-1929-4fb0-82ee-ab6a785ca9e5\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb"
Nov 03 18:14:17.070814 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.070789    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ede64d2a-75d4-4d21-8592-1680fb138e4e-istio-token\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl\" (UID: \"ede64d2a-75d4-4d21-8592-1680fb138e4e\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl"
Nov 03 18:14:17.110243 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.110214    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb-tmp-dir\") pod \"09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb\" (UID: \"09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb\") "
Nov 03 18:14:17.110431 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.110286    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb-kube-api-access-vmbzb\" (UniqueName: \"kubernetes.io/projected/09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb-kube-api-access-vmbzb\") pod \"09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb\" (UID: \"09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb\") "
Nov 03 18:14:17.110431 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.110314    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb-dshm\") pod \"09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb\" (UID: \"09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb\") "
Nov 03 18:14:17.110553 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.110476    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb-home\" (UniqueName: \"kubernetes.io/empty-dir/09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb-home\") pod \"09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb\" (UID: \"09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb\") "
Nov 03 18:14:17.110553 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.110506    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb-tmp-dir" pod "09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb" (UID: "09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:17.110684 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.110593    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb-tls-certs\" (UniqueName: \"kubernetes.io/secret/09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb-tls-certs\") pod \"09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb\" (UID: \"09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb\") "
Nov 03 18:14:17.110684 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.110667    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb-model-cache\") pod \"09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb\" (UID: \"09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb\") "
Nov 03 18:14:17.110946 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.110918    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb-home" pod "09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb" (UID: "09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:17.111499 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.111476    5315 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb-home\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:17.111705 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.111672    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb-model-cache" pod "09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb" (UID: "09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:17.111815 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.111698    5315 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb-tmp-dir\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:17.113803 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.113772    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb-dshm" pod "09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb" (UID: "09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:17.113883 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.113831    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb-kube-api-access-vmbzb" pod "09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb" (UID: "09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb"). InnerVolumeSpecName "kube-api-access-vmbzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:17.113920 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.113884    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb-tls-certs" pod "09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb" (UID: "09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:14:17.178393 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.178346    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl"
Nov 03 18:14:17.183586 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.183560    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb"
Nov 03 18:14:17.216542 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.215465    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmbzb\" (UniqueName: \"kubernetes.io/projected/09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb-kube-api-access-vmbzb\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:17.216542 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.215500    5315 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb-dshm\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:17.216542 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.215527    5315 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb-tls-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:17.216542 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.215541    5315 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb-model-cache\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:17.316181 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.316096    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv"]
Nov 03 18:14:17.326018 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:14:17.325986    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda36c9b31_8ab0_449f_8fb4_87a702ab7abf.slice/crio-60d1b9be4df4c908923dedfc2424f28b6384e35ab6177b137558b23dfcb4b47e WatchSource:0}: Error finding container 60d1b9be4df4c908923dedfc2424f28b6384e35ab6177b137558b23dfcb4b47e: Status 404 returned error can't find the container with id 60d1b9be4df4c908923dedfc2424f28b6384e35ab6177b137558b23dfcb4b47e
Nov 03 18:14:17.329730 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.329685    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:14:17.329864 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.329786    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:14:17.329864 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.329830    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:14:17.393136 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.393101    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv" event={"ID":"a36c9b31-8ab0-449f-8fb4-87a702ab7abf","Type":"ContainerStarted","Data":"60d1b9be4df4c908923dedfc2424f28b6384e35ab6177b137558b23dfcb4b47e"}
Nov 03 18:14:17.395185 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.395158    5315 generic.go:356] "Generic (PLEG): container finished" podID="09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb" containerID="83357fb07c4a01250285ec6a651a109a9f966ad2f35ef294f0cab0100e8cd9ea" exitCode=0
Nov 03 18:14:17.395312 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.395212    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7"
Nov 03 18:14:17.395312 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.395240    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7" event={"ID":"09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb","Type":"ContainerDied","Data":"83357fb07c4a01250285ec6a651a109a9f966ad2f35ef294f0cab0100e8cd9ea"}
Nov 03 18:14:17.395312 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.395288    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7" event={"ID":"09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb","Type":"ContainerDied","Data":"c285ef0b545c5b32b6bd61ee68a761289768d4b04b4b2fd19cf24e9eeb63df3a"}
Nov 03 18:14:17.395442 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.395314    5315 scope.go:122] "RemoveContainer" containerID="83357fb07c4a01250285ec6a651a109a9f966ad2f35ef294f0cab0100e8cd9ea"
Nov 03 18:14:17.441331 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.441304    5315 scope.go:122] "RemoveContainer" containerID="83357fb07c4a01250285ec6a651a109a9f966ad2f35ef294f0cab0100e8cd9ea"
Nov 03 18:14:17.441750 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:14:17.441726    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83357fb07c4a01250285ec6a651a109a9f966ad2f35ef294f0cab0100e8cd9ea\": container with ID starting with 83357fb07c4a01250285ec6a651a109a9f966ad2f35ef294f0cab0100e8cd9ea not found: ID does not exist" containerID="83357fb07c4a01250285ec6a651a109a9f966ad2f35ef294f0cab0100e8cd9ea"
Nov 03 18:14:17.441837 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.441760    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83357fb07c4a01250285ec6a651a109a9f966ad2f35ef294f0cab0100e8cd9ea"} err="failed to get container status \"83357fb07c4a01250285ec6a651a109a9f966ad2f35ef294f0cab0100e8cd9ea\": rpc error: code = NotFound desc = could not find container \"83357fb07c4a01250285ec6a651a109a9f966ad2f35ef294f0cab0100e8cd9ea\": container with ID starting with 83357fb07c4a01250285ec6a651a109a9f966ad2f35ef294f0cab0100e8cd9ea not found: ID does not exist"
Nov 03 18:14:17.550493 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.550458    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7"]
Nov 03 18:14:17.554145 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.554117    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7"]
Nov 03 18:14:17.568810 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.568779    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl"]
Nov 03 18:14:17.578491 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.578449    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:14:17.578602 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.578540    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:14:17.578664 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.578603    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:14:17.679175 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.672405    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb"]
Nov 03 18:14:17.719632 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.713473    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:14:17.719632 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.713578    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:14:17.719632 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.713643    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:14:17.736430 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:17.736372    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-w3-0adcbc-kserve-7bff5644f5-c55z7" podUID="09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:14:18.407982 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:18.407941    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl" event={"ID":"ede64d2a-75d4-4d21-8592-1680fb138e4e","Type":"ContainerStarted","Data":"5d0e5bd47c52ec88bb63cf23f78248dfef2112f33d18d424fce6bb5bead52493"}
Nov 03 18:14:18.408494 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:18.408017    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl" event={"ID":"ede64d2a-75d4-4d21-8592-1680fb138e4e","Type":"ContainerStarted","Data":"b6a5444be3aaf0ebdb403638fc42ee12578aa1cf55735a39e5007afedb4f43a5"}
Nov 03 18:14:18.411350 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:18.411315    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb" event={"ID":"c4428ff6-1929-4fb0-82ee-ab6a785ca9e5","Type":"ContainerStarted","Data":"459e5409eec81d918c9469ae21ec9004f857a88c259bf3ff5bf625f9f3760b59"}
Nov 03 18:14:18.411494 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:18.411376    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb" event={"ID":"c4428ff6-1929-4fb0-82ee-ab6a785ca9e5","Type":"ContainerStarted","Data":"e6719d16c14c7ffbb704b2b57bbcd46a567575fb2389e4ab9eceb8aa6c424380"}
Nov 03 18:14:18.413369 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:18.413330    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv" event={"ID":"a36c9b31-8ab0-449f-8fb4-87a702ab7abf","Type":"ContainerStarted","Data":"416a1054372c9b857a182516290a16c306c50e7e7da97e7c03e19d464ce85f71"}
Nov 03 18:14:18.477787 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:18.477721    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-260dd0-76766ff95-8999v" podUID="232bbb8c-f8b8-42a7-a29c-f288fe7e99f9" containerName="payload-processing" containerID="cri-o://af7a33ccf822429da41bda0717e7a0d2d35d422652f1c35271b2255e5581c33f" gracePeriod=25
Nov 03 18:14:18.523662 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:18.523560    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl" podStartSLOduration=2.52353865 podStartE2EDuration="2.52353865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:14:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:14:18.521694812 +0000 UTC m=+1697.537892897" watchObservedRunningTime="2026-11-03 18:14:18.52353865 +0000 UTC m=+1697.539736720"
Nov 03 18:14:18.575993 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:18.575587    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-260dd0-5469bdccd7-82scv" podUID="05817bd1-81be-4585-b1c2-0f3ca80eec9d" containerName="payload-pre-processing" containerID="cri-o://205edba7c548de87bc5f9f181b6a86b719ad557bfaeb93c540842081ae226b1d" gracePeriod=25
Nov 03 18:14:18.651634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:18.647584    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv" podStartSLOduration=2.6475644640000002 podStartE2EDuration="2.647564464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:14:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:14:18.644959444 +0000 UTC m=+1697.661157548" watchObservedRunningTime="2026-11-03 18:14:18.647564464 +0000 UTC m=+1697.663762535"
Nov 03 18:14:18.822220 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:18.822189    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-260dd0-76766ff95-8999v"
Nov 03 18:14:18.864817 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:18.864788    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv"
Nov 03 18:14:18.865189 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:18.865166    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Nov 03 18:14:18.865279 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:18.865221    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv" podUID="a36c9b31-8ab0-449f-8fb4-87a702ab7abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Nov 03 18:14:18.868135 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:18.868114    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/232bbb8c-f8b8-42a7-a29c-f288fe7e99f9-kube-api-access-bshlb\" (UniqueName: \"kubernetes.io/projected/232bbb8c-f8b8-42a7-a29c-f288fe7e99f9-kube-api-access-bshlb\") pod \"232bbb8c-f8b8-42a7-a29c-f288fe7e99f9\" (UID: \"232bbb8c-f8b8-42a7-a29c-f288fe7e99f9\") "
Nov 03 18:14:18.868638 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:18.868584    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/232bbb8c-f8b8-42a7-a29c-f288fe7e99f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/232bbb8c-f8b8-42a7-a29c-f288fe7e99f9-plugins-config-volume\") pod \"232bbb8c-f8b8-42a7-a29c-f288fe7e99f9\" (UID: \"232bbb8c-f8b8-42a7-a29c-f288fe7e99f9\") "
Nov 03 18:14:18.869070 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:18.869042    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/232bbb8c-f8b8-42a7-a29c-f288fe7e99f9-plugins-config-volume" pod "232bbb8c-f8b8-42a7-a29c-f288fe7e99f9" (UID: "232bbb8c-f8b8-42a7-a29c-f288fe7e99f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:14:18.869863 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:18.869838    5315 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/232bbb8c-f8b8-42a7-a29c-f288fe7e99f9-plugins-config-volume\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:18.871575 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:18.871396    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/232bbb8c-f8b8-42a7-a29c-f288fe7e99f9-kube-api-access-bshlb" pod "232bbb8c-f8b8-42a7-a29c-f288fe7e99f9" (UID: "232bbb8c-f8b8-42a7-a29c-f288fe7e99f9"). InnerVolumeSpecName "kube-api-access-bshlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:18.969600 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:18.969470    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb" podStartSLOduration=2.969448277 podStartE2EDuration="2.969448277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:14:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:14:18.767365591 +0000 UTC m=+1697.783563658" watchObservedRunningTime="2026-11-03 18:14:18.969448277 +0000 UTC m=+1697.985646347"
Nov 03 18:14:18.979048 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:18.979001    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bshlb\" (UniqueName: \"kubernetes.io/projected/232bbb8c-f8b8-42a7-a29c-f288fe7e99f9-kube-api-access-bshlb\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:19.025641 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:19.020455    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-260dd0-5469bdccd7-82scv"
Nov 03 18:14:19.053649 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:19.050847    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-aab753-7bb466b675-bqngj"]
Nov 03 18:14:19.109209 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:19.109173    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb" path="/var/lib/kubelet/pods/09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb/volumes"
Nov 03 18:14:19.184633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:19.180805    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl"
Nov 03 18:14:19.185324 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:19.185286    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05817bd1-81be-4585-b1c2-0f3ca80eec9d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05817bd1-81be-4585-b1c2-0f3ca80eec9d-plugins-config-volume\") pod \"05817bd1-81be-4585-b1c2-0f3ca80eec9d\" (UID: \"05817bd1-81be-4585-b1c2-0f3ca80eec9d\") "
Nov 03 18:14:19.185802 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:19.185775    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05817bd1-81be-4585-b1c2-0f3ca80eec9d-kube-api-access-tqxn4\" (UniqueName: \"kubernetes.io/projected/05817bd1-81be-4585-b1c2-0f3ca80eec9d-kube-api-access-tqxn4\") pod \"05817bd1-81be-4585-b1c2-0f3ca80eec9d\" (UID: \"05817bd1-81be-4585-b1c2-0f3ca80eec9d\") "
Nov 03 18:14:19.194120 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:19.194089    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb"
Nov 03 18:14:19.212799 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:19.201964    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Nov 03 18:14:19.212799 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:19.202029    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl" podUID="ede64d2a-75d4-4d21-8592-1680fb138e4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Nov 03 18:14:19.212799 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:19.202436    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05817bd1-81be-4585-b1c2-0f3ca80eec9d-kube-api-access-tqxn4" pod "05817bd1-81be-4585-b1c2-0f3ca80eec9d" (UID: "05817bd1-81be-4585-b1c2-0f3ca80eec9d"). InnerVolumeSpecName "kube-api-access-tqxn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:19.212799 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:19.206695    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05817bd1-81be-4585-b1c2-0f3ca80eec9d-plugins-config-volume" pod "05817bd1-81be-4585-b1c2-0f3ca80eec9d" (UID: "05817bd1-81be-4585-b1c2-0f3ca80eec9d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:14:19.281520 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:19.250897    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Nov 03 18:14:19.281520 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:19.250961    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb" podUID="c4428ff6-1929-4fb0-82ee-ab6a785ca9e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Nov 03 18:14:19.311629 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:19.299419    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqxn4\" (UniqueName: \"kubernetes.io/projected/05817bd1-81be-4585-b1c2-0f3ca80eec9d-kube-api-access-tqxn4\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:19.311629 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:19.299454    5315 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05817bd1-81be-4585-b1c2-0f3ca80eec9d-plugins-config-volume\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:20.015286 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:20.015228    5315 generic.go:356] "Generic (PLEG): container finished" podID="232bbb8c-f8b8-42a7-a29c-f288fe7e99f9" containerID="af7a33ccf822429da41bda0717e7a0d2d35d422652f1c35271b2255e5581c33f" exitCode=0
Nov 03 18:14:20.016359 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:20.016319    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-260dd0-76766ff95-8999v" event={"ID":"232bbb8c-f8b8-42a7-a29c-f288fe7e99f9","Type":"ContainerDied","Data":"af7a33ccf822429da41bda0717e7a0d2d35d422652f1c35271b2255e5581c33f"}
Nov 03 18:14:20.017444 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:20.017423    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-260dd0-76766ff95-8999v" event={"ID":"232bbb8c-f8b8-42a7-a29c-f288fe7e99f9","Type":"ContainerDied","Data":"572f2e9b439a53ebf257bc6305fdc553b4ed9c6e965c6db1b1ea18bdd7429984"}
Nov 03 18:14:20.017928 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:20.017897    5315 scope.go:122] "RemoveContainer" containerID="af7a33ccf822429da41bda0717e7a0d2d35d422652f1c35271b2255e5581c33f"
Nov 03 18:14:20.018894 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:20.018863    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-260dd0-76766ff95-8999v"
Nov 03 18:14:20.020303 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:20.017016    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Nov 03 18:14:20.020484 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:20.020463    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv" podUID="a36c9b31-8ab0-449f-8fb4-87a702ab7abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Nov 03 18:14:20.081250 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:20.081198    5315 generic.go:356] "Generic (PLEG): container finished" podID="05817bd1-81be-4585-b1c2-0f3ca80eec9d" containerID="205edba7c548de87bc5f9f181b6a86b719ad557bfaeb93c540842081ae226b1d" exitCode=0
Nov 03 18:14:20.083476 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:20.083416    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-260dd0-5469bdccd7-82scv"
Nov 03 18:14:20.084031 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:20.083960    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-260dd0-5469bdccd7-82scv" event={"ID":"05817bd1-81be-4585-b1c2-0f3ca80eec9d","Type":"ContainerDied","Data":"205edba7c548de87bc5f9f181b6a86b719ad557bfaeb93c540842081ae226b1d"}
Nov 03 18:14:20.084143 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:20.084031    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-260dd0-5469bdccd7-82scv" event={"ID":"05817bd1-81be-4585-b1c2-0f3ca80eec9d","Type":"ContainerDied","Data":"9e097ea73994def176bdbd91bd8cb02953c2a9f5cfa1bce789d5144c026db238"}
Nov 03 18:14:20.093000 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:20.084944    5315 scope.go:122] "RemoveContainer" containerID="af7a33ccf822429da41bda0717e7a0d2d35d422652f1c35271b2255e5581c33f"
Nov 03 18:14:20.131972 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:14:20.122785    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af7a33ccf822429da41bda0717e7a0d2d35d422652f1c35271b2255e5581c33f\": container with ID starting with af7a33ccf822429da41bda0717e7a0d2d35d422652f1c35271b2255e5581c33f not found: ID does not exist" containerID="af7a33ccf822429da41bda0717e7a0d2d35d422652f1c35271b2255e5581c33f"
Nov 03 18:14:20.131972 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:20.122853    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af7a33ccf822429da41bda0717e7a0d2d35d422652f1c35271b2255e5581c33f"} err="failed to get container status \"af7a33ccf822429da41bda0717e7a0d2d35d422652f1c35271b2255e5581c33f\": rpc error: code = NotFound desc = could not find container \"af7a33ccf822429da41bda0717e7a0d2d35d422652f1c35271b2255e5581c33f\": container with ID starting with af7a33ccf822429da41bda0717e7a0d2d35d422652f1c35271b2255e5581c33f not found: ID does not exist"
Nov 03 18:14:20.131972 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:20.122882    5315 scope.go:122] "RemoveContainer" containerID="205edba7c548de87bc5f9f181b6a86b719ad557bfaeb93c540842081ae226b1d"
Nov 03 18:14:20.184995 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:20.184956    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Nov 03 18:14:20.186042 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:20.185700    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Nov 03 18:14:20.186042 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:20.185991    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb" podUID="c4428ff6-1929-4fb0-82ee-ab6a785ca9e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Nov 03 18:14:20.186647 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:20.185940    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl" podUID="ede64d2a-75d4-4d21-8592-1680fb138e4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Nov 03 18:14:20.219543 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:20.219518    5315 scope.go:122] "RemoveContainer" containerID="205edba7c548de87bc5f9f181b6a86b719ad557bfaeb93c540842081ae226b1d"
Nov 03 18:14:20.220049 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:14:20.220006    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"205edba7c548de87bc5f9f181b6a86b719ad557bfaeb93c540842081ae226b1d\": container with ID starting with 205edba7c548de87bc5f9f181b6a86b719ad557bfaeb93c540842081ae226b1d not found: ID does not exist" containerID="205edba7c548de87bc5f9f181b6a86b719ad557bfaeb93c540842081ae226b1d"
Nov 03 18:14:20.220154 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:20.220059    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"205edba7c548de87bc5f9f181b6a86b719ad557bfaeb93c540842081ae226b1d"} err="failed to get container status \"205edba7c548de87bc5f9f181b6a86b719ad557bfaeb93c540842081ae226b1d\": rpc error: code = NotFound desc = could not find container \"205edba7c548de87bc5f9f181b6a86b719ad557bfaeb93c540842081ae226b1d\": container with ID starting with 205edba7c548de87bc5f9f181b6a86b719ad557bfaeb93c540842081ae226b1d not found: ID does not exist"
Nov 03 18:14:20.415158 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:20.415131    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-260dd0-76766ff95-8999v"]
Nov 03 18:14:20.419533 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:20.419504    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-260dd0-76766ff95-8999v"]
Nov 03 18:14:20.586711 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:20.586665    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-260dd0-5469bdccd7-82scv"]
Nov 03 18:14:20.616635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:20.603751    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-260dd0-5469bdccd7-82scv"]
Nov 03 18:14:20.876634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:20.867143    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Nov 03 18:14:20.876634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:20.867203    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv" podUID="a36c9b31-8ab0-449f-8fb4-87a702ab7abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Nov 03 18:14:21.119811 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:21.119747    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-0adcbc-695b8c899-b9rjk" podUID="0fa9bc33-97f0-40eb-aee3-f2a6923736a1" containerName="payload-processing" containerID="cri-o://fc5e8915c7cb2bf3e174c352cbe61e150bfd3ed046ea85bf4309f4fba8df2739" gracePeriod=25
Nov 03 18:14:21.123897 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:21.123834    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05817bd1-81be-4585-b1c2-0f3ca80eec9d" path="/var/lib/kubelet/pods/05817bd1-81be-4585-b1c2-0f3ca80eec9d/volumes"
Nov 03 18:14:21.124801 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:21.124775    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="232bbb8c-f8b8-42a7-a29c-f288fe7e99f9" path="/var/lib/kubelet/pods/232bbb8c-f8b8-42a7-a29c-f288fe7e99f9/volumes"
Nov 03 18:14:21.190632 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:21.180132    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Nov 03 18:14:21.190632 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:21.180225    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl" podUID="ede64d2a-75d4-4d21-8592-1680fb138e4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Nov 03 18:14:21.190632 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:21.184054    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Nov 03 18:14:21.190632 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:21.184103    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb" podUID="c4428ff6-1929-4fb0-82ee-ab6a785ca9e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Nov 03 18:14:21.233654 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:21.232122    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0adcbc-b4854959-pwnvs" podUID="1120dc4d-3916-41fc-99b4-3c91ff69f28e" containerName="payload-pre-processing" containerID="cri-o://5c2775dd53a7b83d1599b3c4d2042d589d9d72bb66ef5047e99a05378e6638b1" gracePeriod=25
Nov 03 18:14:21.867652 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:21.864228    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Nov 03 18:14:21.867652 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:21.864287    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv" podUID="a36c9b31-8ab0-449f-8fb4-87a702ab7abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Nov 03 18:14:22.181984 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:22.181882    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Nov 03 18:14:22.182574 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:22.182537    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl" podUID="ede64d2a-75d4-4d21-8592-1680fb138e4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Nov 03 18:14:22.188057 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:22.188025    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Nov 03 18:14:22.188372 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:22.188342    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb" podUID="c4428ff6-1929-4fb0-82ee-ab6a785ca9e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Nov 03 18:14:22.398638 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:22.397831    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37b4fe53-tb2hn"]
Nov 03 18:14:22.398909 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:22.398861    5315 cpu_manager.go:414] "containerMap: removing container" podUID="232bbb8c-f8b8-42a7-a29c-f288fe7e99f9" containerName="payload-processing"
Nov 03 18:14:22.398909 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:22.398891    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="232bbb8c-f8b8-42a7-a29c-f288fe7e99f9" containerName="payload-processing"
Nov 03 18:14:22.399058 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:22.398946    5315 cpu_manager.go:414] "containerMap: removing container" podUID="05817bd1-81be-4585-b1c2-0f3ca80eec9d" containerName="payload-pre-processing"
Nov 03 18:14:22.399058 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:22.398956    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05817bd1-81be-4585-b1c2-0f3ca80eec9d" containerName="payload-pre-processing"
Nov 03 18:14:22.399058 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:22.398985    5315 cpu_manager.go:414] "containerMap: removing container" podUID="09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb" containerName="main"
Nov 03 18:14:22.399058 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:22.398992    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb" containerName="main"
Nov 03 18:14:22.399278 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:22.399156    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="09cc9cc8-2f5e-49e8-8134-1e9deec0d6fb" containerName="main"
Nov 03 18:14:22.399278 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:22.399178    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="232bbb8c-f8b8-42a7-a29c-f288fe7e99f9" containerName="payload-processing"
Nov 03 18:14:22.399278 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:22.399197    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="05817bd1-81be-4585-b1c2-0f3ca80eec9d" containerName="payload-pre-processing"
Nov 03 18:14:22.405309 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:22.405284    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b4fe53-tb2hn"
Nov 03 18:14:22.420396 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:22.420347    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dff4788-m9mbb"]
Nov 03 18:14:22.433645 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:22.433556    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dff4788-m9mbb"
Nov 03 18:14:22.457765 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:22.457736    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:14:22.459765 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:22.458143    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:14:22.459932 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:22.459674    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37b4fe53-tb2hn\" (UID: \"b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b4fe53-tb2hn"
Nov 03 18:14:22.460146 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:22.460126    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37b4fe53-tb2hn\" (UID: \"b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b4fe53-tb2hn"
Nov 03 18:14:22.460323 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:22.460308    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvkkg\" (UniqueName: \"kubernetes.io/projected/b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af-kube-api-access-xvkkg\") pod \"maas-api-revoke-sub-37b4fe53-tb2hn\" (UID: \"b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b4fe53-tb2hn"
Nov 03 18:14:22.480188 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:22.480157    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:14:22.481750 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:22.480432    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:14:22.503742 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:22.503712    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37b4fe53-tb2hn"]
Nov 03 18:14:22.603337 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:22.603283    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37b4fe53-tb2hn\" (UID: \"b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b4fe53-tb2hn"
Nov 03 18:14:22.603562 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:22.603369    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/959ef213-8356-4ab2-8649-0378e33730d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dff4788-m9mbb\" (UID: \"959ef213-8356-4ab2-8649-0378e33730d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dff4788-m9mbb"
Nov 03 18:14:22.603562 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:22.603471    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvkkg\" (UniqueName: \"kubernetes.io/projected/b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af-kube-api-access-xvkkg\") pod \"maas-api-revoke-sub-37b4fe53-tb2hn\" (UID: \"b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b4fe53-tb2hn"
Nov 03 18:14:22.603562 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:22.603515    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/959ef213-8356-4ab2-8649-0378e33730d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dff4788-m9mbb\" (UID: \"959ef213-8356-4ab2-8649-0378e33730d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dff4788-m9mbb"
Nov 03 18:14:22.603782 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:22.603588    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjbmd\" (UniqueName: \"kubernetes.io/projected/959ef213-8356-4ab2-8649-0378e33730d5-kube-api-access-tjbmd\") pod \"maas-api-revoke-sub-1dff4788-m9mbb\" (UID: \"959ef213-8356-4ab2-8649-0378e33730d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dff4788-m9mbb"
Nov 03 18:14:22.606144 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:22.605633    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37b4fe53-tb2hn\" (UID: \"b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b4fe53-tb2hn"
Nov 03 18:14:22.613067 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:22.613030    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37b4fe53-tb2hn\" (UID: \"b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b4fe53-tb2hn"
Nov 03 18:14:22.620096 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:22.620068    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37b4fe53-tb2hn\" (UID: \"b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b4fe53-tb2hn"
Nov 03 18:14:22.705705 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:22.701714    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dff4788-m9mbb"]
Nov 03 18:14:22.715349 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:22.715317    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvkkg\" (UniqueName: \"kubernetes.io/projected/b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af-kube-api-access-xvkkg\") pod \"maas-api-revoke-sub-37b4fe53-tb2hn\" (UID: \"b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b4fe53-tb2hn"
Nov 03 18:14:22.720803 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:22.720779    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b4fe53-tb2hn"
Nov 03 18:14:22.721742 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:22.721715    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/959ef213-8356-4ab2-8649-0378e33730d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dff4788-m9mbb\" (UID: \"959ef213-8356-4ab2-8649-0378e33730d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dff4788-m9mbb"
Nov 03 18:14:22.722053 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:22.722034    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/959ef213-8356-4ab2-8649-0378e33730d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dff4788-m9mbb\" (UID: \"959ef213-8356-4ab2-8649-0378e33730d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dff4788-m9mbb"
Nov 03 18:14:22.722231 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:22.722217    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjbmd\" (UniqueName: \"kubernetes.io/projected/959ef213-8356-4ab2-8649-0378e33730d5-kube-api-access-tjbmd\") pod \"maas-api-revoke-sub-1dff4788-m9mbb\" (UID: \"959ef213-8356-4ab2-8649-0378e33730d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dff4788-m9mbb"
Nov 03 18:14:22.799688 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:22.761699    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/959ef213-8356-4ab2-8649-0378e33730d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dff4788-m9mbb\" (UID: \"959ef213-8356-4ab2-8649-0378e33730d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dff4788-m9mbb"
Nov 03 18:14:22.809704 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:22.809592    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjbmd\" (UniqueName: \"kubernetes.io/projected/959ef213-8356-4ab2-8649-0378e33730d5-kube-api-access-tjbmd\") pod \"maas-api-revoke-sub-1dff4788-m9mbb\" (UID: \"959ef213-8356-4ab2-8649-0378e33730d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dff4788-m9mbb"
Nov 03 18:14:22.814670 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:22.814635    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/959ef213-8356-4ab2-8649-0378e33730d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dff4788-m9mbb\" (UID: \"959ef213-8356-4ab2-8649-0378e33730d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dff4788-m9mbb"
Nov 03 18:14:22.887774 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:22.886808    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Nov 03 18:14:22.887774 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:22.886873    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv" podUID="a36c9b31-8ab0-449f-8fb4-87a702ab7abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Nov 03 18:14:23.106226 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:23.106180    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dff4788-m9mbb"
Nov 03 18:14:23.217701 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:23.213947    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Nov 03 18:14:23.217701 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:23.214006    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl" podUID="ede64d2a-75d4-4d21-8592-1680fb138e4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Nov 03 18:14:23.217701 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:23.214605    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Nov 03 18:14:23.217701 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:23.214654    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb" podUID="c4428ff6-1929-4fb0-82ee-ab6a785ca9e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Nov 03 18:14:23.906716 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:23.894701    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Nov 03 18:14:23.906716 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:23.894764    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv" podUID="a36c9b31-8ab0-449f-8fb4-87a702ab7abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Nov 03 18:14:24.087813 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:24.085111    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aab753-7bb466b675-bqngj" podUID="0f010c6f-b9a7-4367-874e-3412af41460a" containerName="payload-pre-processing" containerID="cri-o://1d8a1abb27f3003b5561b1517d2b000e5586fefe9fa1ab7b4b8ae640166de107" gracePeriod=25
Nov 03 18:14:24.182302 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:24.182168    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37b4fe53-tb2hn"]
Nov 03 18:14:24.361697 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:24.285799    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Nov 03 18:14:24.361697 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:24.285861    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl" podUID="ede64d2a-75d4-4d21-8592-1680fb138e4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Nov 03 18:14:24.361697 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:24.286281    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-232f9e-75b58576ff-2z4jb"]
Nov 03 18:14:24.361697 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:24.340752    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Nov 03 18:14:24.361697 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:24.340830    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb" podUID="c4428ff6-1929-4fb0-82ee-ab6a785ca9e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Nov 03 18:14:24.361697 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:24.344985    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l" podUID="114dcf54-e4eb-4294-a77b-604fe6d57d48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:14:24.554121 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:24.554077    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7" podUID="33725a51-6459-4f15-a866-e9dee41a9732" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:14:24.648635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:24.647515    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-232f9e-75b58576ff-2z4jb"
Nov 03 18:14:24.689642 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:14:24.688564    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:14:24.707255 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:24.707198    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-232f9e\""
Nov 03 18:14:24.708080 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:24.708031    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-232f9e-dockercfg-f68xn\""
Nov 03 18:14:24.730747 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:24.719668    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr" podUID="08159a02-fc2e-4138-824d-be5ba3e141d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:14:24.771569 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:24.771530    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-232f9e-75b58576ff-2z4jb"]
Nov 03 18:14:24.793581 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:24.773136    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s6g9\" (UniqueName: \"kubernetes.io/projected/0a25b9cd-06ba-4a5f-b73f-e63177585ff2-kube-api-access-6s6g9\") pod \"payload-processing-e2e-shared-b-w5-232f9e-75b58576ff-2z4jb\" (UID: \"0a25b9cd-06ba-4a5f-b73f-e63177585ff2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-232f9e-75b58576ff-2z4jb"
Nov 03 18:14:24.794039 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:24.794001    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a25b9cd-06ba-4a5f-b73f-e63177585ff2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-232f9e-75b58576ff-2z4jb\" (UID: \"0a25b9cd-06ba-4a5f-b73f-e63177585ff2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-232f9e-75b58576ff-2z4jb"
Nov 03 18:14:24.811060 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:24.810345    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-232f9e-649f6fd75-thwmg"]
Nov 03 18:14:24.836689 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:24.830546    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-232f9e-649f6fd75-thwmg"
Nov 03 18:14:24.836902 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:24.836878    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-232f9e-649f6fd75-thwmg"]
Nov 03 18:14:24.914920 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:24.914837    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54n6g\" (UniqueName: \"kubernetes.io/projected/c6ca8679-f0c2-48a0-8786-ed4cbad5295d-kube-api-access-54n6g\") pod \"payload-pre-processing-e2e-shared-b-w5-232f9e-649f6fd75-thwmg\" (UID: \"c6ca8679-f0c2-48a0-8786-ed4cbad5295d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-232f9e-649f6fd75-thwmg"
Nov 03 18:14:24.915318 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:24.915296    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s6g9\" (UniqueName: \"kubernetes.io/projected/0a25b9cd-06ba-4a5f-b73f-e63177585ff2-kube-api-access-6s6g9\") pod \"payload-processing-e2e-shared-b-w5-232f9e-75b58576ff-2z4jb\" (UID: \"0a25b9cd-06ba-4a5f-b73f-e63177585ff2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-232f9e-75b58576ff-2z4jb"
Nov 03 18:14:24.915504 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:24.915490    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a25b9cd-06ba-4a5f-b73f-e63177585ff2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-232f9e-75b58576ff-2z4jb\" (UID: \"0a25b9cd-06ba-4a5f-b73f-e63177585ff2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-232f9e-75b58576ff-2z4jb"
Nov 03 18:14:24.915772 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:24.915751    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6ca8679-f0c2-48a0-8786-ed4cbad5295d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-232f9e-649f6fd75-thwmg\" (UID: \"c6ca8679-f0c2-48a0-8786-ed4cbad5295d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-232f9e-649f6fd75-thwmg"
Nov 03 18:14:24.916093 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:24.915043    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Nov 03 18:14:24.916233 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:24.916211    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv" podUID="a36c9b31-8ab0-449f-8fb4-87a702ab7abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Nov 03 18:14:24.923404 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:24.920890    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dff4788-m9mbb"]
Nov 03 18:14:24.923563 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:24.923515    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a25b9cd-06ba-4a5f-b73f-e63177585ff2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-232f9e-75b58576ff-2z4jb\" (UID: \"0a25b9cd-06ba-4a5f-b73f-e63177585ff2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-232f9e-75b58576ff-2z4jb"
Nov 03 18:14:25.008431 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:25.008399    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s6g9\" (UniqueName: \"kubernetes.io/projected/0a25b9cd-06ba-4a5f-b73f-e63177585ff2-kube-api-access-6s6g9\") pod \"payload-processing-e2e-shared-b-w5-232f9e-75b58576ff-2z4jb\" (UID: \"0a25b9cd-06ba-4a5f-b73f-e63177585ff2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-232f9e-75b58576ff-2z4jb"
Nov 03 18:14:25.014028 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:25.014003    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-232f9e-75b58576ff-2z4jb"
Nov 03 18:14:25.055406 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:25.055276    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6ca8679-f0c2-48a0-8786-ed4cbad5295d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-232f9e-649f6fd75-thwmg\" (UID: \"c6ca8679-f0c2-48a0-8786-ed4cbad5295d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-232f9e-649f6fd75-thwmg"
Nov 03 18:14:25.056163 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:25.056045    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54n6g\" (UniqueName: \"kubernetes.io/projected/c6ca8679-f0c2-48a0-8786-ed4cbad5295d-kube-api-access-54n6g\") pod \"payload-pre-processing-e2e-shared-b-w5-232f9e-649f6fd75-thwmg\" (UID: \"c6ca8679-f0c2-48a0-8786-ed4cbad5295d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-232f9e-649f6fd75-thwmg"
Nov 03 18:14:25.060077 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:25.060046    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6ca8679-f0c2-48a0-8786-ed4cbad5295d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-232f9e-649f6fd75-thwmg\" (UID: \"c6ca8679-f0c2-48a0-8786-ed4cbad5295d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-232f9e-649f6fd75-thwmg"
Nov 03 18:14:25.173731 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:25.161344    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54n6g\" (UniqueName: \"kubernetes.io/projected/c6ca8679-f0c2-48a0-8786-ed4cbad5295d-kube-api-access-54n6g\") pod \"payload-pre-processing-e2e-shared-b-w5-232f9e-649f6fd75-thwmg\" (UID: \"c6ca8679-f0c2-48a0-8786-ed4cbad5295d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-232f9e-649f6fd75-thwmg"
Nov 03 18:14:25.201816 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:25.201772    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Nov 03 18:14:25.202069 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:25.201859    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl" podUID="ede64d2a-75d4-4d21-8592-1680fb138e4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Nov 03 18:14:25.209406 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:25.209140    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Nov 03 18:14:25.209406 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:25.209221    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb" podUID="c4428ff6-1929-4fb0-82ee-ab6a785ca9e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Nov 03 18:14:25.411082 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:25.411017    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-232f9e-649f6fd75-thwmg"
Nov 03 18:14:25.605224 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:25.605179    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dff4788-m9mbb" event={"ID":"959ef213-8356-4ab2-8649-0378e33730d5","Type":"ContainerStarted","Data":"7979bc31f2e3230cb6a459359c495abb5feb4ac9ac9820979a2d8149f86b7508"}
Nov 03 18:14:25.608072 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:25.608010    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dff4788-m9mbb" event={"ID":"959ef213-8356-4ab2-8649-0378e33730d5","Type":"ContainerStarted","Data":"9125fe456ab853e939d5d8157e99e2964addcee23327f06335157921a04900be"}
Nov 03 18:14:25.609654 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:25.609451    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b4fe53-tb2hn" event={"ID":"b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af","Type":"ContainerStarted","Data":"3a39b8e8c5e24ba7d67e14d0be941370f95cfdfe6965fe249f80ba249e1c15c3"}
Nov 03 18:14:25.609654 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:25.609499    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b4fe53-tb2hn" event={"ID":"b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af","Type":"ContainerStarted","Data":"59e059aecbdbcc263f57cdf3cda3ba430947370b6bcaa4e505c352c929b067aa"}
Nov 03 18:14:25.802636 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:25.788040    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dff4788-m9mbb" podStartSLOduration=3.788018459 podStartE2EDuration="3.788018459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:14:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:14:25.749210874 +0000 UTC m=+1704.765408955" watchObservedRunningTime="2026-11-03 18:14:25.788018459 +0000 UTC m=+1704.804216530"
Nov 03 18:14:25.879769 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:25.866447    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Nov 03 18:14:25.879769 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:25.866518    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv" podUID="a36c9b31-8ab0-449f-8fb4-87a702ab7abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Nov 03 18:14:25.896106 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:25.896082    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-232f9e-75b58576ff-2z4jb"]
Nov 03 18:14:25.898665 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:25.897123    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b4fe53-tb2hn" podStartSLOduration=3.897106896 podStartE2EDuration="3.897106896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:14:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:14:25.866108677 +0000 UTC m=+1704.882306747" watchObservedRunningTime="2026-11-03 18:14:25.897106896 +0000 UTC m=+1704.913304966"
Nov 03 18:14:26.179416 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:26.179373    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Nov 03 18:14:26.179596 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:26.179436    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl" podUID="ede64d2a-75d4-4d21-8592-1680fb138e4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Nov 03 18:14:26.184899 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:26.184867    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Nov 03 18:14:26.185032 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:26.184917    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb" podUID="c4428ff6-1929-4fb0-82ee-ab6a785ca9e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Nov 03 18:14:26.223690 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:26.223652    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-232f9e-649f6fd75-thwmg"]
Nov 03 18:14:26.289832 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:14:26.289789    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6ca8679_f0c2_48a0_8786_ed4cbad5295d.slice/crio-ad0103ce399d3f5719ab60c8dbc51e62cbe461078fb4512bd3ef32289989976b WatchSource:0}: Error finding container ad0103ce399d3f5719ab60c8dbc51e62cbe461078fb4512bd3ef32289989976b: Status 404 returned error can't find the container with id ad0103ce399d3f5719ab60c8dbc51e62cbe461078fb4512bd3ef32289989976b
Nov 03 18:14:26.463989 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:26.463939    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aab753-7bb466b675-bqngj" podUID="0f010c6f-b9a7-4367-874e-3412af41460a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:14:26.623101 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:26.623014    5315 generic.go:356] "Generic (PLEG): container finished" podID="b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af" containerID="3a39b8e8c5e24ba7d67e14d0be941370f95cfdfe6965fe249f80ba249e1c15c3" exitCode=0
Nov 03 18:14:26.623641 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:26.623583    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b4fe53-tb2hn" event={"ID":"b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af","Type":"ContainerDied","Data":"3a39b8e8c5e24ba7d67e14d0be941370f95cfdfe6965fe249f80ba249e1c15c3"}
Nov 03 18:14:26.625768 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:26.625739    5315 generic.go:356] "Generic (PLEG): container finished" podID="959ef213-8356-4ab2-8649-0378e33730d5" containerID="7979bc31f2e3230cb6a459359c495abb5feb4ac9ac9820979a2d8149f86b7508" exitCode=0
Nov 03 18:14:26.625899 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:26.625801    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dff4788-m9mbb" event={"ID":"959ef213-8356-4ab2-8649-0378e33730d5","Type":"ContainerDied","Data":"7979bc31f2e3230cb6a459359c495abb5feb4ac9ac9820979a2d8149f86b7508"}
Nov 03 18:14:26.628008 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:26.627979    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-232f9e-649f6fd75-thwmg" event={"ID":"c6ca8679-f0c2-48a0-8786-ed4cbad5295d","Type":"ContainerStarted","Data":"8366f8c386c20c9a31fd816b8f379cabd66ae38b01939c41eb7ea15da0cfaab6"}
Nov 03 18:14:26.628135 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:26.628036    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-232f9e-649f6fd75-thwmg" event={"ID":"c6ca8679-f0c2-48a0-8786-ed4cbad5295d","Type":"ContainerStarted","Data":"ad0103ce399d3f5719ab60c8dbc51e62cbe461078fb4512bd3ef32289989976b"}
Nov 03 18:14:26.628135 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:26.628073    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-232f9e-649f6fd75-thwmg"
Nov 03 18:14:26.630299 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:26.630262    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-232f9e-75b58576ff-2z4jb" event={"ID":"0a25b9cd-06ba-4a5f-b73f-e63177585ff2","Type":"ContainerStarted","Data":"67b2b09bdc9091fac78291aadbe5b9786299a10f208a965b93cbbeb07130fb3e"}
Nov 03 18:14:26.630474 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:26.630312    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-232f9e-75b58576ff-2z4jb" event={"ID":"0a25b9cd-06ba-4a5f-b73f-e63177585ff2","Type":"ContainerStarted","Data":"b44c730079131456d7a678672fe7bd6c530a5e815b512532e5160aae053309b6"}
Nov 03 18:14:26.630474 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:26.630385    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-232f9e-75b58576ff-2z4jb"
Nov 03 18:14:26.705508 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:26.705448    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv" podUID="f6d706cb-3aa5-419c-8f6c-29347bb788b1" containerName="main" containerID="cri-o://6b51db0b64221ebe2f34ab8d9b57bc9508616ed4b1c71fca16944f6476a6461c" gracePeriod=45
Nov 03 18:14:26.712834 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:26.712792    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7" podUID="33725a51-6459-4f15-a866-e9dee41a9732" containerName="main" containerID="cri-o://f615d5f66f36b3600db5973901433e72ea6cf322a41c20ee33c0772f7662e910" gracePeriod=45
Nov 03 18:14:26.720813 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:26.720778    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv" podUID="f6d706cb-3aa5-419c-8f6c-29347bb788b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": dial tcp 10.133.0.45:8000: connect: connection refused"
Nov 03 18:14:26.843966 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:26.843185    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-232f9e-649f6fd75-thwmg" podStartSLOduration=2.843165247 podStartE2EDuration="2.843165247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:14:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:14:26.839319837 +0000 UTC m=+1705.855517918" watchObservedRunningTime="2026-11-03 18:14:26.843165247 +0000 UTC m=+1705.859363317"
Nov 03 18:14:27.149430 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.131484    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-232f9e-75b58576ff-2z4jb" podStartSLOduration=3.131460832 podStartE2EDuration="3.131460832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:14:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:14:27.128662521 +0000 UTC m=+1706.144860592" watchObservedRunningTime="2026-11-03 18:14:27.131460832 +0000 UTC m=+1706.147658901"
Nov 03 18:14:27.243865 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.243828    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Nov 03 18:14:27.244090 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.244059    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl" podUID="ede64d2a-75d4-4d21-8592-1680fb138e4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Nov 03 18:14:27.318756 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.318245    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv"
Nov 03 18:14:27.322554 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.322518    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7"
Nov 03 18:14:27.471727 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.468227    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6d706cb-3aa5-419c-8f6c-29347bb788b1-home\" (UniqueName: \"kubernetes.io/empty-dir/f6d706cb-3aa5-419c-8f6c-29347bb788b1-home\") pod \"f6d706cb-3aa5-419c-8f6c-29347bb788b1\" (UID: \"f6d706cb-3aa5-419c-8f6c-29347bb788b1\") "
Nov 03 18:14:27.471727 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.468290    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6d706cb-3aa5-419c-8f6c-29347bb788b1-tls-certs\" (UniqueName: \"kubernetes.io/secret/f6d706cb-3aa5-419c-8f6c-29347bb788b1-tls-certs\") pod \"f6d706cb-3aa5-419c-8f6c-29347bb788b1\" (UID: \"f6d706cb-3aa5-419c-8f6c-29347bb788b1\") "
Nov 03 18:14:27.471727 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.468343    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33725a51-6459-4f15-a866-e9dee41a9732-home\" (UniqueName: \"kubernetes.io/empty-dir/33725a51-6459-4f15-a866-e9dee41a9732-home\") pod \"33725a51-6459-4f15-a866-e9dee41a9732\" (UID: \"33725a51-6459-4f15-a866-e9dee41a9732\") "
Nov 03 18:14:27.471727 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.468437    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33725a51-6459-4f15-a866-e9dee41a9732-kube-api-access-blvks\" (UniqueName: \"kubernetes.io/projected/33725a51-6459-4f15-a866-e9dee41a9732-kube-api-access-blvks\") pod \"33725a51-6459-4f15-a866-e9dee41a9732\" (UID: \"33725a51-6459-4f15-a866-e9dee41a9732\") "
Nov 03 18:14:27.471727 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.468516    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33725a51-6459-4f15-a866-e9dee41a9732-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33725a51-6459-4f15-a866-e9dee41a9732-model-cache\") pod \"33725a51-6459-4f15-a866-e9dee41a9732\" (UID: \"33725a51-6459-4f15-a866-e9dee41a9732\") "
Nov 03 18:14:27.471727 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.468658    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6d706cb-3aa5-419c-8f6c-29347bb788b1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6d706cb-3aa5-419c-8f6c-29347bb788b1-model-cache\") pod \"f6d706cb-3aa5-419c-8f6c-29347bb788b1\" (UID: \"f6d706cb-3aa5-419c-8f6c-29347bb788b1\") "
Nov 03 18:14:27.471727 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.468754    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33725a51-6459-4f15-a866-e9dee41a9732-tls-certs\" (UniqueName: \"kubernetes.io/secret/33725a51-6459-4f15-a866-e9dee41a9732-tls-certs\") pod \"33725a51-6459-4f15-a866-e9dee41a9732\" (UID: \"33725a51-6459-4f15-a866-e9dee41a9732\") "
Nov 03 18:14:27.471727 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.468789    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6d706cb-3aa5-419c-8f6c-29347bb788b1-kube-api-access-zqvb5\" (UniqueName: \"kubernetes.io/projected/f6d706cb-3aa5-419c-8f6c-29347bb788b1-kube-api-access-zqvb5\") pod \"f6d706cb-3aa5-419c-8f6c-29347bb788b1\" (UID: \"f6d706cb-3aa5-419c-8f6c-29347bb788b1\") "
Nov 03 18:14:27.471727 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.468897    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6d706cb-3aa5-419c-8f6c-29347bb788b1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6d706cb-3aa5-419c-8f6c-29347bb788b1-tmp-dir\") pod \"f6d706cb-3aa5-419c-8f6c-29347bb788b1\" (UID: \"f6d706cb-3aa5-419c-8f6c-29347bb788b1\") "
Nov 03 18:14:27.471727 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.469000    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33725a51-6459-4f15-a866-e9dee41a9732-dshm\" (UniqueName: \"kubernetes.io/empty-dir/33725a51-6459-4f15-a866-e9dee41a9732-dshm\") pod \"33725a51-6459-4f15-a866-e9dee41a9732\" (UID: \"33725a51-6459-4f15-a866-e9dee41a9732\") "
Nov 03 18:14:27.471727 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.469033    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6d706cb-3aa5-419c-8f6c-29347bb788b1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6d706cb-3aa5-419c-8f6c-29347bb788b1-dshm\") pod \"f6d706cb-3aa5-419c-8f6c-29347bb788b1\" (UID: \"f6d706cb-3aa5-419c-8f6c-29347bb788b1\") "
Nov 03 18:14:27.471727 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.469096    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33725a51-6459-4f15-a866-e9dee41a9732-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33725a51-6459-4f15-a866-e9dee41a9732-tmp-dir\") pod \"33725a51-6459-4f15-a866-e9dee41a9732\" (UID: \"33725a51-6459-4f15-a866-e9dee41a9732\") "
Nov 03 18:14:27.487647 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.481924    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6d706cb-3aa5-419c-8f6c-29347bb788b1-home" pod "f6d706cb-3aa5-419c-8f6c-29347bb788b1" (UID: "f6d706cb-3aa5-419c-8f6c-29347bb788b1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:27.487647 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.482698    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6d706cb-3aa5-419c-8f6c-29347bb788b1-tmp-dir" pod "f6d706cb-3aa5-419c-8f6c-29347bb788b1" (UID: "f6d706cb-3aa5-419c-8f6c-29347bb788b1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:27.492731 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.492693    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33725a51-6459-4f15-a866-e9dee41a9732-home" pod "33725a51-6459-4f15-a866-e9dee41a9732" (UID: "33725a51-6459-4f15-a866-e9dee41a9732"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:27.493146 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.493118    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6d706cb-3aa5-419c-8f6c-29347bb788b1-model-cache" pod "f6d706cb-3aa5-419c-8f6c-29347bb788b1" (UID: "f6d706cb-3aa5-419c-8f6c-29347bb788b1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:27.509447 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.496083    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33725a51-6459-4f15-a866-e9dee41a9732-model-cache" pod "33725a51-6459-4f15-a866-e9dee41a9732" (UID: "33725a51-6459-4f15-a866-e9dee41a9732"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:27.509665 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.509457    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33725a51-6459-4f15-a866-e9dee41a9732-tmp-dir" pod "33725a51-6459-4f15-a866-e9dee41a9732" (UID: "33725a51-6459-4f15-a866-e9dee41a9732"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:27.510580 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.510542    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33725a51-6459-4f15-a866-e9dee41a9732-tls-certs" pod "33725a51-6459-4f15-a866-e9dee41a9732" (UID: "33725a51-6459-4f15-a866-e9dee41a9732"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:14:27.510769 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.510743    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6d706cb-3aa5-419c-8f6c-29347bb788b1-kube-api-access-zqvb5" pod "f6d706cb-3aa5-419c-8f6c-29347bb788b1" (UID: "f6d706cb-3aa5-419c-8f6c-29347bb788b1"). InnerVolumeSpecName "kube-api-access-zqvb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:27.511215 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.511186    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6d706cb-3aa5-419c-8f6c-29347bb788b1-dshm" pod "f6d706cb-3aa5-419c-8f6c-29347bb788b1" (UID: "f6d706cb-3aa5-419c-8f6c-29347bb788b1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:27.511444 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.511401    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6d706cb-3aa5-419c-8f6c-29347bb788b1-tls-certs" pod "f6d706cb-3aa5-419c-8f6c-29347bb788b1" (UID: "f6d706cb-3aa5-419c-8f6c-29347bb788b1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:14:27.520640 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.519762    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33725a51-6459-4f15-a866-e9dee41a9732-dshm" pod "33725a51-6459-4f15-a866-e9dee41a9732" (UID: "33725a51-6459-4f15-a866-e9dee41a9732"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:27.521416 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.521372    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33725a51-6459-4f15-a866-e9dee41a9732-kube-api-access-blvks" pod "33725a51-6459-4f15-a866-e9dee41a9732" (UID: "33725a51-6459-4f15-a866-e9dee41a9732"). InnerVolumeSpecName "kube-api-access-blvks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:27.635990 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.635292    5315 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6d706cb-3aa5-419c-8f6c-29347bb788b1-model-cache\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:27.635990 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.635336    5315 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33725a51-6459-4f15-a866-e9dee41a9732-tls-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:27.635990 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.635356    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqvb5\" (UniqueName: \"kubernetes.io/projected/f6d706cb-3aa5-419c-8f6c-29347bb788b1-kube-api-access-zqvb5\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:27.635990 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.635372    5315 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6d706cb-3aa5-419c-8f6c-29347bb788b1-tmp-dir\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:27.635990 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.635386    5315 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33725a51-6459-4f15-a866-e9dee41a9732-dshm\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:27.635990 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.635400    5315 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6d706cb-3aa5-419c-8f6c-29347bb788b1-dshm\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:27.635990 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.635413    5315 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33725a51-6459-4f15-a866-e9dee41a9732-tmp-dir\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:27.635990 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.635426    5315 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6d706cb-3aa5-419c-8f6c-29347bb788b1-home\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:27.635990 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.635440    5315 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6d706cb-3aa5-419c-8f6c-29347bb788b1-tls-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:27.635990 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.635452    5315 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33725a51-6459-4f15-a866-e9dee41a9732-home\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:27.635990 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.635466    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blvks\" (UniqueName: \"kubernetes.io/projected/33725a51-6459-4f15-a866-e9dee41a9732-kube-api-access-blvks\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:27.635990 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.635481    5315 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33725a51-6459-4f15-a866-e9dee41a9732-model-cache\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:27.653647 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.650775    5315 generic.go:356] "Generic (PLEG): container finished" podID="f6d706cb-3aa5-419c-8f6c-29347bb788b1" containerID="6b51db0b64221ebe2f34ab8d9b57bc9508616ed4b1c71fca16944f6476a6461c" exitCode=0
Nov 03 18:14:27.653647 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.650934    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv" event={"ID":"f6d706cb-3aa5-419c-8f6c-29347bb788b1","Type":"ContainerDied","Data":"6b51db0b64221ebe2f34ab8d9b57bc9508616ed4b1c71fca16944f6476a6461c"}
Nov 03 18:14:27.653647 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.650976    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv" event={"ID":"f6d706cb-3aa5-419c-8f6c-29347bb788b1","Type":"ContainerDied","Data":"4deaeda3db0ad2b7f4f35533006d02feca793df378ce6752519f321eda4c8715"}
Nov 03 18:14:27.653647 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.651000    5315 scope.go:122] "RemoveContainer" containerID="6b51db0b64221ebe2f34ab8d9b57bc9508616ed4b1c71fca16944f6476a6461c"
Nov 03 18:14:27.653647 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.651154    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv"
Nov 03 18:14:27.660147 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.660099    5315 generic.go:356] "Generic (PLEG): container finished" podID="33725a51-6459-4f15-a866-e9dee41a9732" containerID="f615d5f66f36b3600db5973901433e72ea6cf322a41c20ee33c0772f7662e910" exitCode=0
Nov 03 18:14:27.662364 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.662143    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7" event={"ID":"33725a51-6459-4f15-a866-e9dee41a9732","Type":"ContainerDied","Data":"f615d5f66f36b3600db5973901433e72ea6cf322a41c20ee33c0772f7662e910"}
Nov 03 18:14:27.663639 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.663573    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7" event={"ID":"33725a51-6459-4f15-a866-e9dee41a9732","Type":"ContainerDied","Data":"f0501a2c6ba816fc0b669d00ecfb67e0f6f224458b290b5cb9fab2df65eeb95c"}
Nov 03 18:14:27.664451 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.664045    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7"
Nov 03 18:14:27.744320 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.741648    5315 scope.go:122] "RemoveContainer" containerID="6b51db0b64221ebe2f34ab8d9b57bc9508616ed4b1c71fca16944f6476a6461c"
Nov 03 18:14:27.744320 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:14:27.742626    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b51db0b64221ebe2f34ab8d9b57bc9508616ed4b1c71fca16944f6476a6461c\": container with ID starting with 6b51db0b64221ebe2f34ab8d9b57bc9508616ed4b1c71fca16944f6476a6461c not found: ID does not exist" containerID="6b51db0b64221ebe2f34ab8d9b57bc9508616ed4b1c71fca16944f6476a6461c"
Nov 03 18:14:27.744320 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.742665    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b51db0b64221ebe2f34ab8d9b57bc9508616ed4b1c71fca16944f6476a6461c"} err="failed to get container status \"6b51db0b64221ebe2f34ab8d9b57bc9508616ed4b1c71fca16944f6476a6461c\": rpc error: code = NotFound desc = could not find container \"6b51db0b64221ebe2f34ab8d9b57bc9508616ed4b1c71fca16944f6476a6461c\": container with ID starting with 6b51db0b64221ebe2f34ab8d9b57bc9508616ed4b1c71fca16944f6476a6461c not found: ID does not exist"
Nov 03 18:14:27.744320 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.742690    5315 scope.go:122] "RemoveContainer" containerID="f615d5f66f36b3600db5973901433e72ea6cf322a41c20ee33c0772f7662e910"
Nov 03 18:14:27.801118 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.800658    5315 scope.go:122] "RemoveContainer" containerID="f615d5f66f36b3600db5973901433e72ea6cf322a41c20ee33c0772f7662e910"
Nov 03 18:14:27.801217 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:14:27.801171    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f615d5f66f36b3600db5973901433e72ea6cf322a41c20ee33c0772f7662e910\": container with ID starting with f615d5f66f36b3600db5973901433e72ea6cf322a41c20ee33c0772f7662e910 not found: ID does not exist" containerID="f615d5f66f36b3600db5973901433e72ea6cf322a41c20ee33c0772f7662e910"
Nov 03 18:14:27.801290 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.801212    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f615d5f66f36b3600db5973901433e72ea6cf322a41c20ee33c0772f7662e910"} err="failed to get container status \"f615d5f66f36b3600db5973901433e72ea6cf322a41c20ee33c0772f7662e910\": rpc error: code = NotFound desc = could not find container \"f615d5f66f36b3600db5973901433e72ea6cf322a41c20ee33c0772f7662e910\": container with ID starting with f615d5f66f36b3600db5973901433e72ea6cf322a41c20ee33c0772f7662e910 not found: ID does not exist"
Nov 03 18:14:27.900267 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.826945    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7"]
Nov 03 18:14:27.900267 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.834658    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-aab753/simulator-w6-aab753-kserve-79c4b67bbb-h6rx7"]
Nov 03 18:14:27.900267 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.881632    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:14:27.900267 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.881687    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv" podUID="a36c9b31-8ab0-449f-8fb4-87a702ab7abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:14:27.927629 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.927556    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv"
Nov 03 18:14:27.946162 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.929554    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv"
Nov 03 18:14:27.956926 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.956379    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv"]
Nov 03 18:14:27.961476 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.961445    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-aab753/premium-w6-aab753-kserve-56975cd5c9-fsrtv"]
Nov 03 18:14:27.967140 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:27.964010    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv"
Nov 03 18:14:28.147953 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:28.129578    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb"
Nov 03 18:14:28.147953 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:28.131364    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb"
Nov 03 18:14:28.147953 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:28.135397    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb"
Nov 03 18:14:28.149042 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:28.148857    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b4fe53-tb2hn"
Nov 03 18:14:28.155362 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:28.155335    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dff4788-m9mbb"
Nov 03 18:14:28.217415 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:28.217383    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af-kube-api-access-xvkkg\" (UniqueName: \"kubernetes.io/projected/b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af-kube-api-access-xvkkg\") pod \"b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af\" (UID: \"b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af\") "
Nov 03 18:14:28.217578 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:28.217469    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/959ef213-8356-4ab2-8649-0378e33730d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/959ef213-8356-4ab2-8649-0378e33730d5-maas-api-openshift-ca\") pod \"959ef213-8356-4ab2-8649-0378e33730d5\" (UID: \"959ef213-8356-4ab2-8649-0378e33730d5\") "
Nov 03 18:14:28.217578 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:28.217555    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af-maas-api-openshift-ca\") pod \"b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af\" (UID: \"b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af\") "
Nov 03 18:14:28.217743 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:28.217635    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af-maas-api-xks-ca\") pod \"b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af\" (UID: \"b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af\") "
Nov 03 18:14:28.217743 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:28.217663    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/959ef213-8356-4ab2-8649-0378e33730d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/959ef213-8356-4ab2-8649-0378e33730d5-maas-api-xks-ca\") pod \"959ef213-8356-4ab2-8649-0378e33730d5\" (UID: \"959ef213-8356-4ab2-8649-0378e33730d5\") "
Nov 03 18:14:28.217852 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:28.217829    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/959ef213-8356-4ab2-8649-0378e33730d5-kube-api-access-tjbmd\" (UniqueName: \"kubernetes.io/projected/959ef213-8356-4ab2-8649-0378e33730d5-kube-api-access-tjbmd\") pod \"959ef213-8356-4ab2-8649-0378e33730d5\" (UID: \"959ef213-8356-4ab2-8649-0378e33730d5\") "
Nov 03 18:14:28.218016 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:28.217990    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/959ef213-8356-4ab2-8649-0378e33730d5-maas-api-openshift-ca" pod "959ef213-8356-4ab2-8649-0378e33730d5" (UID: "959ef213-8356-4ab2-8649-0378e33730d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:14:28.218089 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:28.218015    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af-maas-api-openshift-ca" pod "b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af" (UID: "b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:14:28.218590 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:28.218533    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/959ef213-8356-4ab2-8649-0378e33730d5-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:28.218590 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:28.218565    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:28.220736 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:28.220706    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af-maas-api-xks-ca" pod "b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af" (UID: "b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:14:28.221159 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:28.221120    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af-kube-api-access-xvkkg" pod "b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af" (UID: "b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af"). InnerVolumeSpecName "kube-api-access-xvkkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:28.221325 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:28.221285    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/959ef213-8356-4ab2-8649-0378e33730d5-kube-api-access-tjbmd" pod "959ef213-8356-4ab2-8649-0378e33730d5" (UID: "959ef213-8356-4ab2-8649-0378e33730d5"). InnerVolumeSpecName "kube-api-access-tjbmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:28.221503 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:28.221466    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/959ef213-8356-4ab2-8649-0378e33730d5-maas-api-xks-ca" pod "959ef213-8356-4ab2-8649-0378e33730d5" (UID: "959ef213-8356-4ab2-8649-0378e33730d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:14:28.320569 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:28.320519    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:28.320569 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:28.320566    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/959ef213-8356-4ab2-8649-0378e33730d5-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:28.320885 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:28.320582    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjbmd\" (UniqueName: \"kubernetes.io/projected/959ef213-8356-4ab2-8649-0378e33730d5-kube-api-access-tjbmd\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:28.320885 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:28.320598    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvkkg\" (UniqueName: \"kubernetes.io/projected/b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af-kube-api-access-xvkkg\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:28.323893 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:28.323706    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl"
Nov 03 18:14:28.324554 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:28.324482    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl"
Nov 03 18:14:28.326044 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:28.326009    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl"
Nov 03 18:14:28.671750 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:28.671645    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b4fe53-tb2hn" event={"ID":"b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af","Type":"ContainerDied","Data":"59e059aecbdbcc263f57cdf3cda3ba430947370b6bcaa4e505c352c929b067aa"}
Nov 03 18:14:28.671750 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:28.671714    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59e059aecbdbcc263f57cdf3cda3ba430947370b6bcaa4e505c352c929b067aa"
Nov 03 18:14:28.671750 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:28.671667    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b4fe53-tb2hn"
Nov 03 18:14:28.674734 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:28.674699    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dff4788-m9mbb"
Nov 03 18:14:28.674901 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:28.674737    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dff4788-m9mbb" event={"ID":"959ef213-8356-4ab2-8649-0378e33730d5","Type":"ContainerDied","Data":"9125fe456ab853e939d5d8157e99e2964addcee23327f06335157921a04900be"}
Nov 03 18:14:28.674901 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:28.674793    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9125fe456ab853e939d5d8157e99e2964addcee23327f06335157921a04900be"
Nov 03 18:14:29.109859 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:29.109824    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33725a51-6459-4f15-a866-e9dee41a9732" path="/var/lib/kubelet/pods/33725a51-6459-4f15-a866-e9dee41a9732/volumes"
Nov 03 18:14:29.110454 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:29.110431    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6d706cb-3aa5-419c-8f6c-29347bb788b1" path="/var/lib/kubelet/pods/f6d706cb-3aa5-419c-8f6c-29347bb788b1/volumes"
Nov 03 18:14:29.585776 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:29.585739    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0adcbc-b4854959-pwnvs" podUID="1120dc4d-3916-41fc-99b4-3c91ff69f28e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:14:29.585978 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:29.585743    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-0adcbc-695b8c899-b9rjk" podUID="0fa9bc33-97f0-40eb-aee3-f2a6923736a1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:14:36.462338 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:36.462295    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aab753-7bb466b675-bqngj" podUID="0f010c6f-b9a7-4367-874e-3412af41460a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:14:37.667451 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:37.667416    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-232f9e-649f6fd75-thwmg"
Nov 03 18:14:37.670819 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:37.670795    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-232f9e-75b58576ff-2z4jb"
Nov 03 18:14:39.584473 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:39.584434    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0adcbc-b4854959-pwnvs" podUID="1120dc4d-3916-41fc-99b4-3c91ff69f28e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:14:39.584929 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:39.584433    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-0adcbc-695b8c899-b9rjk" podUID="0fa9bc33-97f0-40eb-aee3-f2a6923736a1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:14:41.439292 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:41.439259    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-0adcbc-695b8c899-b9rjk"
Nov 03 18:14:41.489140 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:41.489102    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fa9bc33-97f0-40eb-aee3-f2a6923736a1-kube-api-access-snq2h\" (UniqueName: \"kubernetes.io/projected/0fa9bc33-97f0-40eb-aee3-f2a6923736a1-kube-api-access-snq2h\") pod \"0fa9bc33-97f0-40eb-aee3-f2a6923736a1\" (UID: \"0fa9bc33-97f0-40eb-aee3-f2a6923736a1\") "
Nov 03 18:14:41.489372 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:41.489352    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fa9bc33-97f0-40eb-aee3-f2a6923736a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fa9bc33-97f0-40eb-aee3-f2a6923736a1-plugins-config-volume\") pod \"0fa9bc33-97f0-40eb-aee3-f2a6923736a1\" (UID: \"0fa9bc33-97f0-40eb-aee3-f2a6923736a1\") "
Nov 03 18:14:41.491135 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:41.490825    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fa9bc33-97f0-40eb-aee3-f2a6923736a1-plugins-config-volume" pod "0fa9bc33-97f0-40eb-aee3-f2a6923736a1" (UID: "0fa9bc33-97f0-40eb-aee3-f2a6923736a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:14:41.493515 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:41.493471    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fa9bc33-97f0-40eb-aee3-f2a6923736a1-kube-api-access-snq2h" pod "0fa9bc33-97f0-40eb-aee3-f2a6923736a1" (UID: "0fa9bc33-97f0-40eb-aee3-f2a6923736a1"). InnerVolumeSpecName "kube-api-access-snq2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:41.607733 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:41.606201    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snq2h\" (UniqueName: \"kubernetes.io/projected/0fa9bc33-97f0-40eb-aee3-f2a6923736a1-kube-api-access-snq2h\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:41.607733 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:41.606240    5315 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fa9bc33-97f0-40eb-aee3-f2a6923736a1-plugins-config-volume\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:41.630636 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:41.630489    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0adcbc-b4854959-pwnvs"
Nov 03 18:14:41.709688 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:41.708857    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1120dc4d-3916-41fc-99b4-3c91ff69f28e-kube-api-access-ssvsr\" (UniqueName: \"kubernetes.io/projected/1120dc4d-3916-41fc-99b4-3c91ff69f28e-kube-api-access-ssvsr\") pod \"1120dc4d-3916-41fc-99b4-3c91ff69f28e\" (UID: \"1120dc4d-3916-41fc-99b4-3c91ff69f28e\") "
Nov 03 18:14:41.709688 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:41.709087    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1120dc4d-3916-41fc-99b4-3c91ff69f28e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1120dc4d-3916-41fc-99b4-3c91ff69f28e-plugins-config-volume\") pod \"1120dc4d-3916-41fc-99b4-3c91ff69f28e\" (UID: \"1120dc4d-3916-41fc-99b4-3c91ff69f28e\") "
Nov 03 18:14:41.724664 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:41.723250    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1120dc4d-3916-41fc-99b4-3c91ff69f28e-plugins-config-volume" pod "1120dc4d-3916-41fc-99b4-3c91ff69f28e" (UID: "1120dc4d-3916-41fc-99b4-3c91ff69f28e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:14:41.750545 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:41.750486    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1120dc4d-3916-41fc-99b4-3c91ff69f28e-kube-api-access-ssvsr" pod "1120dc4d-3916-41fc-99b4-3c91ff69f28e" (UID: "1120dc4d-3916-41fc-99b4-3c91ff69f28e"). InnerVolumeSpecName "kube-api-access-ssvsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:41.810752 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:41.810716    5315 generic.go:356] "Generic (PLEG): container finished" podID="0fa9bc33-97f0-40eb-aee3-f2a6923736a1" containerID="fc5e8915c7cb2bf3e174c352cbe61e150bfd3ed046ea85bf4309f4fba8df2739" exitCode=0
Nov 03 18:14:41.811163 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:41.811133    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-0adcbc-695b8c899-b9rjk" event={"ID":"0fa9bc33-97f0-40eb-aee3-f2a6923736a1","Type":"ContainerDied","Data":"fc5e8915c7cb2bf3e174c352cbe61e150bfd3ed046ea85bf4309f4fba8df2739"}
Nov 03 18:14:41.811368 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:41.811347    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-0adcbc-695b8c899-b9rjk" event={"ID":"0fa9bc33-97f0-40eb-aee3-f2a6923736a1","Type":"ContainerDied","Data":"f2498e8410ca3f7eceafb27c904828afd6a30d115d5c4acc9f9646a2bbd213b5"}
Nov 03 18:14:41.811551 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:41.811527    5315 scope.go:122] "RemoveContainer" containerID="fc5e8915c7cb2bf3e174c352cbe61e150bfd3ed046ea85bf4309f4fba8df2739"
Nov 03 18:14:41.811896 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:41.811743    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-0adcbc-695b8c899-b9rjk"
Nov 03 18:14:41.819628 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:41.819584    5315 generic.go:356] "Generic (PLEG): container finished" podID="1120dc4d-3916-41fc-99b4-3c91ff69f28e" containerID="5c2775dd53a7b83d1599b3c4d2042d589d9d72bb66ef5047e99a05378e6638b1" exitCode=0
Nov 03 18:14:41.819844 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:41.819806    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0adcbc-b4854959-pwnvs" event={"ID":"1120dc4d-3916-41fc-99b4-3c91ff69f28e","Type":"ContainerDied","Data":"5c2775dd53a7b83d1599b3c4d2042d589d9d72bb66ef5047e99a05378e6638b1"}
Nov 03 18:14:41.820019 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:41.819975    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0adcbc-b4854959-pwnvs" event={"ID":"1120dc4d-3916-41fc-99b4-3c91ff69f28e","Type":"ContainerDied","Data":"4ca0ddfd5f3539c067683518fcfe1eff5bb7c255d6bb5941be8e51bea5af2736"}
Nov 03 18:14:41.820274 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:41.820196    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0adcbc-b4854959-pwnvs"
Nov 03 18:14:41.824803 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:41.824604    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssvsr\" (UniqueName: \"kubernetes.io/projected/1120dc4d-3916-41fc-99b4-3c91ff69f28e-kube-api-access-ssvsr\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:41.824803 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:41.824678    5315 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1120dc4d-3916-41fc-99b4-3c91ff69f28e-plugins-config-volume\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:41.829021 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:41.829000    5315 scope.go:122] "RemoveContainer" containerID="fc5e8915c7cb2bf3e174c352cbe61e150bfd3ed046ea85bf4309f4fba8df2739"
Nov 03 18:14:41.829379 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:14:41.829355    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc5e8915c7cb2bf3e174c352cbe61e150bfd3ed046ea85bf4309f4fba8df2739\": container with ID starting with fc5e8915c7cb2bf3e174c352cbe61e150bfd3ed046ea85bf4309f4fba8df2739 not found: ID does not exist" containerID="fc5e8915c7cb2bf3e174c352cbe61e150bfd3ed046ea85bf4309f4fba8df2739"
Nov 03 18:14:41.829461 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:41.829388    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc5e8915c7cb2bf3e174c352cbe61e150bfd3ed046ea85bf4309f4fba8df2739"} err="failed to get container status \"fc5e8915c7cb2bf3e174c352cbe61e150bfd3ed046ea85bf4309f4fba8df2739\": rpc error: code = NotFound desc = could not find container \"fc5e8915c7cb2bf3e174c352cbe61e150bfd3ed046ea85bf4309f4fba8df2739\": container with ID starting with fc5e8915c7cb2bf3e174c352cbe61e150bfd3ed046ea85bf4309f4fba8df2739 not found: ID does not exist"
Nov 03 18:14:41.829461 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:41.829412    5315 scope.go:122] "RemoveContainer" containerID="5c2775dd53a7b83d1599b3c4d2042d589d9d72bb66ef5047e99a05378e6638b1"
Nov 03 18:14:41.843168 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:41.843095    5315 scope.go:122] "RemoveContainer" containerID="5c2775dd53a7b83d1599b3c4d2042d589d9d72bb66ef5047e99a05378e6638b1"
Nov 03 18:14:41.843534 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:14:41.843510    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c2775dd53a7b83d1599b3c4d2042d589d9d72bb66ef5047e99a05378e6638b1\": container with ID starting with 5c2775dd53a7b83d1599b3c4d2042d589d9d72bb66ef5047e99a05378e6638b1 not found: ID does not exist" containerID="5c2775dd53a7b83d1599b3c4d2042d589d9d72bb66ef5047e99a05378e6638b1"
Nov 03 18:14:41.843702 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:41.843543    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c2775dd53a7b83d1599b3c4d2042d589d9d72bb66ef5047e99a05378e6638b1"} err="failed to get container status \"5c2775dd53a7b83d1599b3c4d2042d589d9d72bb66ef5047e99a05378e6638b1\": rpc error: code = NotFound desc = could not find container \"5c2775dd53a7b83d1599b3c4d2042d589d9d72bb66ef5047e99a05378e6638b1\": container with ID starting with 5c2775dd53a7b83d1599b3c4d2042d589d9d72bb66ef5047e99a05378e6638b1 not found: ID does not exist"
Nov 03 18:14:41.970265 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:41.970111    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-0adcbc-695b8c899-b9rjk"]
Nov 03 18:14:41.973872 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:41.973837    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-0adcbc-695b8c899-b9rjk"]
Nov 03 18:14:42.073279 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:42.073236    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-0adcbc-b4854959-pwnvs"]
Nov 03 18:14:42.077343 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:42.077311    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-0adcbc-b4854959-pwnvs"]
Nov 03 18:14:43.105043 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:43.105004    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fa9bc33-97f0-40eb-aee3-f2a6923736a1" path="/var/lib/kubelet/pods/0fa9bc33-97f0-40eb-aee3-f2a6923736a1/volumes"
Nov 03 18:14:43.105553 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:43.105537    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1120dc4d-3916-41fc-99b4-3c91ff69f28e" path="/var/lib/kubelet/pods/1120dc4d-3916-41fc-99b4-3c91ff69f28e/volumes"
Nov 03 18:14:44.454516 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:44.454484    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aab753-7bb466b675-bqngj"
Nov 03 18:14:44.579877 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:44.579845    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f010c6f-b9a7-4367-874e-3412af41460a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f010c6f-b9a7-4367-874e-3412af41460a-plugins-config-volume\") pod \"0f010c6f-b9a7-4367-874e-3412af41460a\" (UID: \"0f010c6f-b9a7-4367-874e-3412af41460a\") "
Nov 03 18:14:44.580060 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:44.579916    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f010c6f-b9a7-4367-874e-3412af41460a-kube-api-access-rnwj5\" (UniqueName: \"kubernetes.io/projected/0f010c6f-b9a7-4367-874e-3412af41460a-kube-api-access-rnwj5\") pod \"0f010c6f-b9a7-4367-874e-3412af41460a\" (UID: \"0f010c6f-b9a7-4367-874e-3412af41460a\") "
Nov 03 18:14:44.580281 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:44.580256    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f010c6f-b9a7-4367-874e-3412af41460a-plugins-config-volume" pod "0f010c6f-b9a7-4367-874e-3412af41460a" (UID: "0f010c6f-b9a7-4367-874e-3412af41460a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:14:44.580603 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:44.580586    5315 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f010c6f-b9a7-4367-874e-3412af41460a-plugins-config-volume\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:44.582350 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:44.582323    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f010c6f-b9a7-4367-874e-3412af41460a-kube-api-access-rnwj5" pod "0f010c6f-b9a7-4367-874e-3412af41460a" (UID: "0f010c6f-b9a7-4367-874e-3412af41460a"). InnerVolumeSpecName "kube-api-access-rnwj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:44.682058 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:44.682007    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnwj5\" (UniqueName: \"kubernetes.io/projected/0f010c6f-b9a7-4367-874e-3412af41460a-kube-api-access-rnwj5\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:44.854119 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:44.854021    5315 generic.go:356] "Generic (PLEG): container finished" podID="0f010c6f-b9a7-4367-874e-3412af41460a" containerID="1d8a1abb27f3003b5561b1517d2b000e5586fefe9fa1ab7b4b8ae640166de107" exitCode=0
Nov 03 18:14:44.854119 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:44.854064    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aab753-7bb466b675-bqngj"
Nov 03 18:14:44.854119 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:44.854100    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aab753-7bb466b675-bqngj" event={"ID":"0f010c6f-b9a7-4367-874e-3412af41460a","Type":"ContainerDied","Data":"1d8a1abb27f3003b5561b1517d2b000e5586fefe9fa1ab7b4b8ae640166de107"}
Nov 03 18:14:44.854379 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:44.854161    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aab753-7bb466b675-bqngj" event={"ID":"0f010c6f-b9a7-4367-874e-3412af41460a","Type":"ContainerDied","Data":"6038efba68d91282160c4d173c67ad2fd241313642234915b8dabf8f406f9876"}
Nov 03 18:14:44.854379 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:44.854185    5315 scope.go:122] "RemoveContainer" containerID="1d8a1abb27f3003b5561b1517d2b000e5586fefe9fa1ab7b4b8ae640166de107"
Nov 03 18:14:44.865260 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:44.865232    5315 scope.go:122] "RemoveContainer" containerID="1d8a1abb27f3003b5561b1517d2b000e5586fefe9fa1ab7b4b8ae640166de107"
Nov 03 18:14:44.865545 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:14:44.865525    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d8a1abb27f3003b5561b1517d2b000e5586fefe9fa1ab7b4b8ae640166de107\": container with ID starting with 1d8a1abb27f3003b5561b1517d2b000e5586fefe9fa1ab7b4b8ae640166de107 not found: ID does not exist" containerID="1d8a1abb27f3003b5561b1517d2b000e5586fefe9fa1ab7b4b8ae640166de107"
Nov 03 18:14:44.865595 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:44.865555    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d8a1abb27f3003b5561b1517d2b000e5586fefe9fa1ab7b4b8ae640166de107"} err="failed to get container status \"1d8a1abb27f3003b5561b1517d2b000e5586fefe9fa1ab7b4b8ae640166de107\": rpc error: code = NotFound desc = could not find container \"1d8a1abb27f3003b5561b1517d2b000e5586fefe9fa1ab7b4b8ae640166de107\": container with ID starting with 1d8a1abb27f3003b5561b1517d2b000e5586fefe9fa1ab7b4b8ae640166de107 not found: ID does not exist"
Nov 03 18:14:45.007417 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:45.007376    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-aab753-7bb466b675-bqngj"]
Nov 03 18:14:45.011838 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:45.011812    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-aab753-7bb466b675-bqngj"]
Nov 03 18:14:45.104987 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:45.104893    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f010c6f-b9a7-4367-874e-3412af41460a" path="/var/lib/kubelet/pods/0f010c6f-b9a7-4367-874e-3412af41460a/volumes"
Nov 03 18:14:45.562483 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:45.562440    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4"]
Nov 03 18:14:45.562980 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:45.562763    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4" podUID="c3b8cf40-02df-499d-b41c-6e4bd9283a3b" containerName="istio-proxy" containerID="cri-o://94994b1046d894c179ec978ed84492241f9b524e40aaf36f07a96d48c0bbcf72" gracePeriod=30
Nov 03 18:14:45.787637 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:45.786050    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69"]
Nov 03 18:14:45.787637 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:45.786305    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69" podUID="a1bcb2fe-aef1-446a-9e93-a18dc824d388" containerName="istio-proxy" containerID="cri-o://0328b4a91e6c4a00505ad05fdbb246b65a375392a88f17f99f08423cef3b220c" gracePeriod=30
Nov 03 18:14:46.105709 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:46.105285    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs"]
Nov 03 18:14:46.105709 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:46.105515    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs" podUID="1977c777-abc0-4d10-a0ee-3985dd2be5be" containerName="istio-proxy" containerID="cri-o://20c9b60fc479b95e3bf4ceb30bf0668e9fd0cb63001a00378e9709e988fe2cd1" gracePeriod=30
Nov 03 18:14:46.399556 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:46.399447    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m"]
Nov 03 18:14:46.399787 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:46.399740    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m" podUID="df589ab9-3107-4b55-9195-6ee315bd3965" containerName="istio-proxy" containerID="cri-o://1b8d55022722af293a4b45330a88ed9187b8c288ed428fc844fb82dca87afc04" gracePeriod=30
Nov 03 18:14:47.297858 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.297799    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w"]
Nov 03 18:14:47.299405 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.299375    5315 cpu_manager.go:414] "containerMap: removing container" podUID="959ef213-8356-4ab2-8649-0378e33730d5" containerName="revoke-keys"
Nov 03 18:14:47.299579 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.299563    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="959ef213-8356-4ab2-8649-0378e33730d5" containerName="revoke-keys"
Nov 03 18:14:47.299732 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.299717    5315 cpu_manager.go:414] "containerMap: removing container" podUID="0f010c6f-b9a7-4367-874e-3412af41460a" containerName="payload-pre-processing"
Nov 03 18:14:47.299841 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.299828    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f010c6f-b9a7-4367-874e-3412af41460a" containerName="payload-pre-processing"
Nov 03 18:14:47.299934 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.299924    5315 cpu_manager.go:414] "containerMap: removing container" podUID="0fa9bc33-97f0-40eb-aee3-f2a6923736a1" containerName="payload-processing"
Nov 03 18:14:47.300023 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.300011    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fa9bc33-97f0-40eb-aee3-f2a6923736a1" containerName="payload-processing"
Nov 03 18:14:47.300134 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.300122    5315 cpu_manager.go:414] "containerMap: removing container" podUID="33725a51-6459-4f15-a866-e9dee41a9732" containerName="main"
Nov 03 18:14:47.300227 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.300215    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33725a51-6459-4f15-a866-e9dee41a9732" containerName="main"
Nov 03 18:14:47.300331 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.300320    5315 cpu_manager.go:414] "containerMap: removing container" podUID="1120dc4d-3916-41fc-99b4-3c91ff69f28e" containerName="payload-pre-processing"
Nov 03 18:14:47.300408 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.300398    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1120dc4d-3916-41fc-99b4-3c91ff69f28e" containerName="payload-pre-processing"
Nov 03 18:14:47.300509 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.300497    5315 cpu_manager.go:414] "containerMap: removing container" podUID="f6d706cb-3aa5-419c-8f6c-29347bb788b1" containerName="main"
Nov 03 18:14:47.300604 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.300577    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6d706cb-3aa5-419c-8f6c-29347bb788b1" containerName="main"
Nov 03 18:14:47.300736 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.300723    5315 cpu_manager.go:414] "containerMap: removing container" podUID="b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af" containerName="revoke-keys"
Nov 03 18:14:47.300971 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.300954    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af" containerName="revoke-keys"
Nov 03 18:14:47.301277 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.301258    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f010c6f-b9a7-4367-874e-3412af41460a" containerName="payload-pre-processing"
Nov 03 18:14:47.301393 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.301381    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fa9bc33-97f0-40eb-aee3-f2a6923736a1" containerName="payload-processing"
Nov 03 18:14:47.301498 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.301484    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="33725a51-6459-4f15-a866-e9dee41a9732" containerName="main"
Nov 03 18:14:47.301597 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.301586    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="1120dc4d-3916-41fc-99b4-3c91ff69f28e" containerName="payload-pre-processing"
Nov 03 18:14:47.301725 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.301712    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6d706cb-3aa5-419c-8f6c-29347bb788b1" containerName="main"
Nov 03 18:14:47.301808 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.301797    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af" containerName="revoke-keys"
Nov 03 18:14:47.301911 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.301899    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="959ef213-8356-4ab2-8649-0378e33730d5" containerName="revoke-keys"
Nov 03 18:14:47.308144 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.308121    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w"
Nov 03 18:14:47.312986 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.312956    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d07e31d-20b5-47a1-901c-915565686286-istio-envoy\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w\" (UID: \"3d07e31d-20b5-47a1-901c-915565686286\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w"
Nov 03 18:14:47.313189 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.313172    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d07e31d-20b5-47a1-901c-915565686286-workload-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w\" (UID: \"3d07e31d-20b5-47a1-901c-915565686286\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w"
Nov 03 18:14:47.313306 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.313293    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d07e31d-20b5-47a1-901c-915565686286-istio-token\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w\" (UID: \"3d07e31d-20b5-47a1-901c-915565686286\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w"
Nov 03 18:14:47.313480 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.313455    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d07e31d-20b5-47a1-901c-915565686286-istiod-ca-cert\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w\" (UID: \"3d07e31d-20b5-47a1-901c-915565686286\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w"
Nov 03 18:14:47.313554 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.313490    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d07e31d-20b5-47a1-901c-915565686286-istio-data\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w\" (UID: \"3d07e31d-20b5-47a1-901c-915565686286\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w"
Nov 03 18:14:47.313554 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.313534    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrshg\" (UniqueName: \"kubernetes.io/projected/3d07e31d-20b5-47a1-901c-915565686286-kube-api-access-nrshg\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w\" (UID: \"3d07e31d-20b5-47a1-901c-915565686286\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w"
Nov 03 18:14:47.313690 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.313594    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d07e31d-20b5-47a1-901c-915565686286-credential-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w\" (UID: \"3d07e31d-20b5-47a1-901c-915565686286\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w"
Nov 03 18:14:47.313690 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.313672    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d07e31d-20b5-47a1-901c-915565686286-workload-certs\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w\" (UID: \"3d07e31d-20b5-47a1-901c-915565686286\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w"
Nov 03 18:14:47.313796 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.313756    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d07e31d-20b5-47a1-901c-915565686286-istio-podinfo\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w\" (UID: \"3d07e31d-20b5-47a1-901c-915565686286\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w"
Nov 03 18:14:47.334630 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.326096    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg"]
Nov 03 18:14:47.334630 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.332091    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg"
Nov 03 18:14:47.356630 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.349016    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w"]
Nov 03 18:14:47.382647 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.368779    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-0adcbc-openshift-default-dockercfg-zctsw\""
Nov 03 18:14:47.398663 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.396494    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg"]
Nov 03 18:14:47.433643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.417471    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daf3798f-da45-4287-94f6-eb32897e807b-istio-data\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg\" (UID: \"daf3798f-da45-4287-94f6-eb32897e807b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg"
Nov 03 18:14:47.433643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.417544    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d07e31d-20b5-47a1-901c-915565686286-istio-podinfo\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w\" (UID: \"3d07e31d-20b5-47a1-901c-915565686286\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w"
Nov 03 18:14:47.433643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.417654    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d07e31d-20b5-47a1-901c-915565686286-istio-envoy\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w\" (UID: \"3d07e31d-20b5-47a1-901c-915565686286\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w"
Nov 03 18:14:47.433643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.417706    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf3798f-da45-4287-94f6-eb32897e807b-credential-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg\" (UID: \"daf3798f-da45-4287-94f6-eb32897e807b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg"
Nov 03 18:14:47.433643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.417748    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d07e31d-20b5-47a1-901c-915565686286-workload-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w\" (UID: \"3d07e31d-20b5-47a1-901c-915565686286\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w"
Nov 03 18:14:47.433643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.417776    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d07e31d-20b5-47a1-901c-915565686286-istio-token\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w\" (UID: \"3d07e31d-20b5-47a1-901c-915565686286\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w"
Nov 03 18:14:47.433643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.417838    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daf3798f-da45-4287-94f6-eb32897e807b-istiod-ca-cert\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg\" (UID: \"daf3798f-da45-4287-94f6-eb32897e807b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg"
Nov 03 18:14:47.433643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.417872    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d07e31d-20b5-47a1-901c-915565686286-istiod-ca-cert\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w\" (UID: \"3d07e31d-20b5-47a1-901c-915565686286\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w"
Nov 03 18:14:47.433643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.417900    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d07e31d-20b5-47a1-901c-915565686286-istio-data\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w\" (UID: \"3d07e31d-20b5-47a1-901c-915565686286\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w"
Nov 03 18:14:47.433643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.417939    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daf3798f-da45-4287-94f6-eb32897e807b-istio-token\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg\" (UID: \"daf3798f-da45-4287-94f6-eb32897e807b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg"
Nov 03 18:14:47.433643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.417969    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daf3798f-da45-4287-94f6-eb32897e807b-istio-envoy\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg\" (UID: \"daf3798f-da45-4287-94f6-eb32897e807b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg"
Nov 03 18:14:47.433643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.418023    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrshg\" (UniqueName: \"kubernetes.io/projected/3d07e31d-20b5-47a1-901c-915565686286-kube-api-access-nrshg\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w\" (UID: \"3d07e31d-20b5-47a1-901c-915565686286\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w"
Nov 03 18:14:47.433643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.418088    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d07e31d-20b5-47a1-901c-915565686286-credential-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w\" (UID: \"3d07e31d-20b5-47a1-901c-915565686286\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w"
Nov 03 18:14:47.433643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.418122    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf3798f-da45-4287-94f6-eb32897e807b-workload-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg\" (UID: \"daf3798f-da45-4287-94f6-eb32897e807b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg"
Nov 03 18:14:47.433643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.418166    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hppr\" (UniqueName: \"kubernetes.io/projected/daf3798f-da45-4287-94f6-eb32897e807b-kube-api-access-6hppr\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg\" (UID: \"daf3798f-da45-4287-94f6-eb32897e807b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg"
Nov 03 18:14:47.433643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.418194    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daf3798f-da45-4287-94f6-eb32897e807b-workload-certs\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg\" (UID: \"daf3798f-da45-4287-94f6-eb32897e807b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg"
Nov 03 18:14:47.433643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.418236    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d07e31d-20b5-47a1-901c-915565686286-workload-certs\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w\" (UID: \"3d07e31d-20b5-47a1-901c-915565686286\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w"
Nov 03 18:14:47.433643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.418291    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daf3798f-da45-4287-94f6-eb32897e807b-istio-podinfo\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg\" (UID: \"daf3798f-da45-4287-94f6-eb32897e807b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg"
Nov 03 18:14:47.433643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.420266    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d07e31d-20b5-47a1-901c-915565686286-istiod-ca-cert\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w\" (UID: \"3d07e31d-20b5-47a1-901c-915565686286\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w"
Nov 03 18:14:47.433643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.420557    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d07e31d-20b5-47a1-901c-915565686286-istio-data\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w\" (UID: \"3d07e31d-20b5-47a1-901c-915565686286\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w"
Nov 03 18:14:47.433643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.421359    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d07e31d-20b5-47a1-901c-915565686286-credential-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w\" (UID: \"3d07e31d-20b5-47a1-901c-915565686286\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w"
Nov 03 18:14:47.433643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.421634    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d07e31d-20b5-47a1-901c-915565686286-workload-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w\" (UID: \"3d07e31d-20b5-47a1-901c-915565686286\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w"
Nov 03 18:14:47.433643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.421904    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d07e31d-20b5-47a1-901c-915565686286-workload-certs\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w\" (UID: \"3d07e31d-20b5-47a1-901c-915565686286\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w"
Nov 03 18:14:47.433643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.423636    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d07e31d-20b5-47a1-901c-915565686286-istio-podinfo\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w\" (UID: \"3d07e31d-20b5-47a1-901c-915565686286\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w"
Nov 03 18:14:47.513146 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.482518    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d07e31d-20b5-47a1-901c-915565686286-istio-envoy\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w\" (UID: \"3d07e31d-20b5-47a1-901c-915565686286\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w"
Nov 03 18:14:47.529652 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.523572    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daf3798f-da45-4287-94f6-eb32897e807b-istio-data\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg\" (UID: \"daf3798f-da45-4287-94f6-eb32897e807b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg"
Nov 03 18:14:47.529652 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.523766    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf3798f-da45-4287-94f6-eb32897e807b-credential-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg\" (UID: \"daf3798f-da45-4287-94f6-eb32897e807b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg"
Nov 03 18:14:47.529652 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.523858    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daf3798f-da45-4287-94f6-eb32897e807b-istiod-ca-cert\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg\" (UID: \"daf3798f-da45-4287-94f6-eb32897e807b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg"
Nov 03 18:14:47.529652 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.523910    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daf3798f-da45-4287-94f6-eb32897e807b-istio-token\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg\" (UID: \"daf3798f-da45-4287-94f6-eb32897e807b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg"
Nov 03 18:14:47.529652 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.523939    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daf3798f-da45-4287-94f6-eb32897e807b-istio-envoy\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg\" (UID: \"daf3798f-da45-4287-94f6-eb32897e807b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg"
Nov 03 18:14:47.529652 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.524028    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf3798f-da45-4287-94f6-eb32897e807b-workload-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg\" (UID: \"daf3798f-da45-4287-94f6-eb32897e807b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg"
Nov 03 18:14:47.529652 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.524070    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hppr\" (UniqueName: \"kubernetes.io/projected/daf3798f-da45-4287-94f6-eb32897e807b-kube-api-access-6hppr\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg\" (UID: \"daf3798f-da45-4287-94f6-eb32897e807b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg"
Nov 03 18:14:47.529652 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.524099    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daf3798f-da45-4287-94f6-eb32897e807b-workload-certs\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg\" (UID: \"daf3798f-da45-4287-94f6-eb32897e807b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg"
Nov 03 18:14:47.529652 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.524170    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daf3798f-da45-4287-94f6-eb32897e807b-istio-podinfo\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg\" (UID: \"daf3798f-da45-4287-94f6-eb32897e807b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg"
Nov 03 18:14:47.600752 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.572886    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daf3798f-da45-4287-94f6-eb32897e807b-istiod-ca-cert\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg\" (UID: \"daf3798f-da45-4287-94f6-eb32897e807b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg"
Nov 03 18:14:47.600752 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.599739    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daf3798f-da45-4287-94f6-eb32897e807b-istio-data\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg\" (UID: \"daf3798f-da45-4287-94f6-eb32897e807b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg"
Nov 03 18:14:47.600752 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.600033    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf3798f-da45-4287-94f6-eb32897e807b-credential-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg\" (UID: \"daf3798f-da45-4287-94f6-eb32897e807b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg"
Nov 03 18:14:47.600752 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.600559    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf3798f-da45-4287-94f6-eb32897e807b-workload-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg\" (UID: \"daf3798f-da45-4287-94f6-eb32897e807b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg"
Nov 03 18:14:47.603783 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.601917    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daf3798f-da45-4287-94f6-eb32897e807b-workload-certs\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg\" (UID: \"daf3798f-da45-4287-94f6-eb32897e807b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg"
Nov 03 18:14:47.612422 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.612242    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daf3798f-da45-4287-94f6-eb32897e807b-istio-envoy\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg\" (UID: \"daf3798f-da45-4287-94f6-eb32897e807b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg"
Nov 03 18:14:47.637781 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.626379    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daf3798f-da45-4287-94f6-eb32897e807b-istio-podinfo\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg\" (UID: \"daf3798f-da45-4287-94f6-eb32897e807b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg"
Nov 03 18:14:47.697634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.697468    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrshg\" (UniqueName: \"kubernetes.io/projected/3d07e31d-20b5-47a1-901c-915565686286-kube-api-access-nrshg\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w\" (UID: \"3d07e31d-20b5-47a1-901c-915565686286\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w"
Nov 03 18:14:47.711754 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.711720    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daf3798f-da45-4287-94f6-eb32897e807b-istio-token\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg\" (UID: \"daf3798f-da45-4287-94f6-eb32897e807b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg"
Nov 03 18:14:47.731406 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.730325    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d07e31d-20b5-47a1-901c-915565686286-istio-token\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w\" (UID: \"3d07e31d-20b5-47a1-901c-915565686286\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w"
Nov 03 18:14:47.813634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.767913    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hppr\" (UniqueName: \"kubernetes.io/projected/daf3798f-da45-4287-94f6-eb32897e807b-kube-api-access-6hppr\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg\" (UID: \"daf3798f-da45-4287-94f6-eb32897e807b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg"
Nov 03 18:14:47.823910 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.823876    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0ccbb9-7tm97"]
Nov 03 18:14:47.839897 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.829678    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0ccbb9-7tm97"]
Nov 03 18:14:47.938928 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.938860    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w"
Nov 03 18:14:47.999974 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:47.999937    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg"
Nov 03 18:14:48.431194 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:48.431147    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w"]
Nov 03 18:14:48.442712 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:48.442395    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:14:48.442712 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:48.442489    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:14:48.442712 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:48.442534    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:14:48.477503 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:48.477470    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg"]
Nov 03 18:14:48.488089 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:14:48.488055    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaf3798f_da45_4287_94f6_eb32897e807b.slice/crio-ff3ccaaf90c624530731dccba5a0c8d9ff1936269c1f7d2a2a473f1b5bdcd712 WatchSource:0}: Error finding container ff3ccaaf90c624530731dccba5a0c8d9ff1936269c1f7d2a2a473f1b5bdcd712: Status 404 returned error can't find the container with id ff3ccaaf90c624530731dccba5a0c8d9ff1936269c1f7d2a2a473f1b5bdcd712
Nov 03 18:14:48.490966 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:48.490934    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:14:48.491050 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:48.491011    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:14:48.491050 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:48.491043    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:14:48.922927 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:48.922886    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg" event={"ID":"daf3798f-da45-4287-94f6-eb32897e807b","Type":"ContainerStarted","Data":"c34d00783b27c609f5b93682395d9b1b6c44e516167c7b85d02064b2cef11dd8"}
Nov 03 18:14:48.923121 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:48.922953    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg" event={"ID":"daf3798f-da45-4287-94f6-eb32897e807b","Type":"ContainerStarted","Data":"ff3ccaaf90c624530731dccba5a0c8d9ff1936269c1f7d2a2a473f1b5bdcd712"}
Nov 03 18:14:48.924877 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:48.924847    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w" event={"ID":"3d07e31d-20b5-47a1-901c-915565686286","Type":"ContainerStarted","Data":"1b3c165beae7fd0146d0d02033bc972b5d841ca2526f2d2bb77ff800f0adb8c9"}
Nov 03 18:14:48.925116 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:48.924898    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w" event={"ID":"3d07e31d-20b5-47a1-901c-915565686286","Type":"ContainerStarted","Data":"80114b976073b431c77e95a5ae67e3a6c3d764f94d6057e552b1e0928100a54b"}
Nov 03 18:14:48.946766 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:48.946729    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w"
Nov 03 18:14:49.000531 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:49.000496    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg"
Nov 03 18:14:49.001722 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:49.001700    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Nov 03 18:14:49.001781 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:49.001752    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg" podUID="daf3798f-da45-4287-94f6-eb32897e807b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Nov 03 18:14:49.072638 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:49.072515    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg" podStartSLOduration=2.072478438 podStartE2EDuration="2.072478438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:14:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:14:49.057291576 +0000 UTC m=+1728.073489644" watchObservedRunningTime="2026-11-03 18:14:49.072478438 +0000 UTC m=+1728.088676510"
Nov 03 18:14:49.106510 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:49.106481    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad347a4e-f710-4482-a326-5e05d12b0dc3" path="/var/lib/kubelet/pods/ad347a4e-f710-4482-a326-5e05d12b0dc3/volumes"
Nov 03 18:14:49.181829 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:49.176503    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w" podStartSLOduration=2.176480179 podStartE2EDuration="2.176480179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:14:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:14:49.169213036 +0000 UTC m=+1728.185411130" watchObservedRunningTime="2026-11-03 18:14:49.176480179 +0000 UTC m=+1728.192678248"
Nov 03 18:14:50.009026 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:50.008881    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Nov 03 18:14:50.009725 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:50.009589    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg" podUID="daf3798f-da45-4287-94f6-eb32897e807b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Nov 03 18:14:50.058050 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:50.058005    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 03 18:14:50.105632 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:50.105105    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w" podUID="3d07e31d-20b5-47a1-901c-915565686286" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 03 18:14:50.179565 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:50.179407    5315 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:14:50.179565 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:50.179479    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4" podUID="c3b8cf40-02df-499d-b41c-6e4bd9283a3b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:14:50.903348 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:50.900770    5315 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 03 18:14:50.903348 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:50.900829    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69" podUID="a1bcb2fe-aef1-446a-9e93-a18dc824d388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 03 18:14:50.985768 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:50.977708    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 03 18:14:50.985768 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:50.977768    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w" podUID="3d07e31d-20b5-47a1-901c-915565686286" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 03 18:14:51.006796 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:51.006578    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Nov 03 18:14:51.006796 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:51.006682    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg" podUID="daf3798f-da45-4287-94f6-eb32897e807b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Nov 03 18:14:51.095135 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:51.095090    5315 generic.go:356] "Generic (PLEG): container finished" podID="c3b8cf40-02df-499d-b41c-6e4bd9283a3b" containerID="94994b1046d894c179ec978ed84492241f9b524e40aaf36f07a96d48c0bbcf72" exitCode=0
Nov 03 18:14:51.095734 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:51.095173    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4" event={"ID":"c3b8cf40-02df-499d-b41c-6e4bd9283a3b","Type":"ContainerDied","Data":"94994b1046d894c179ec978ed84492241f9b524e40aaf36f07a96d48c0bbcf72"}
Nov 03 18:14:51.098453 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:51.098424    5315 generic.go:356] "Generic (PLEG): container finished" podID="a1bcb2fe-aef1-446a-9e93-a18dc824d388" containerID="0328b4a91e6c4a00505ad05fdbb246b65a375392a88f17f99f08423cef3b220c" exitCode=0
Nov 03 18:14:51.104857 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:51.104828    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69" event={"ID":"a1bcb2fe-aef1-446a-9e93-a18dc824d388","Type":"ContainerDied","Data":"0328b4a91e6c4a00505ad05fdbb246b65a375392a88f17f99f08423cef3b220c"}
Nov 03 18:14:51.238941 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:51.238905    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4"
Nov 03 18:14:51.491712 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:51.457339    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-istio-podinfo\") pod \"c3b8cf40-02df-499d-b41c-6e4bd9283a3b\" (UID: \"c3b8cf40-02df-499d-b41c-6e4bd9283a3b\") "
Nov 03 18:14:51.491712 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:51.457451    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-istio-token\" (UniqueName: \"kubernetes.io/projected/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-istio-token\") pod \"c3b8cf40-02df-499d-b41c-6e4bd9283a3b\" (UID: \"c3b8cf40-02df-499d-b41c-6e4bd9283a3b\") "
Nov 03 18:14:51.491712 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:51.457494    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-istiod-ca-cert\") pod \"c3b8cf40-02df-499d-b41c-6e4bd9283a3b\" (UID: \"c3b8cf40-02df-499d-b41c-6e4bd9283a3b\") "
Nov 03 18:14:51.491712 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:51.457575    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-workload-socket\") pod \"c3b8cf40-02df-499d-b41c-6e4bd9283a3b\" (UID: \"c3b8cf40-02df-499d-b41c-6e4bd9283a3b\") "
Nov 03 18:14:51.491712 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:51.457645    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-kube-api-access-hk2mm\" (UniqueName: \"kubernetes.io/projected/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-kube-api-access-hk2mm\") pod \"c3b8cf40-02df-499d-b41c-6e4bd9283a3b\" (UID: \"c3b8cf40-02df-499d-b41c-6e4bd9283a3b\") "
Nov 03 18:14:51.491712 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:51.457783    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-workload-certs\") pod \"c3b8cf40-02df-499d-b41c-6e4bd9283a3b\" (UID: \"c3b8cf40-02df-499d-b41c-6e4bd9283a3b\") "
Nov 03 18:14:51.491712 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:51.457816    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-credential-socket\") pod \"c3b8cf40-02df-499d-b41c-6e4bd9283a3b\" (UID: \"c3b8cf40-02df-499d-b41c-6e4bd9283a3b\") "
Nov 03 18:14:51.491712 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:51.457852    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-istio-data\") pod \"c3b8cf40-02df-499d-b41c-6e4bd9283a3b\" (UID: \"c3b8cf40-02df-499d-b41c-6e4bd9283a3b\") "
Nov 03 18:14:51.491712 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:51.458082    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-istio-envoy\") pod \"c3b8cf40-02df-499d-b41c-6e4bd9283a3b\" (UID: \"c3b8cf40-02df-499d-b41c-6e4bd9283a3b\") "
Nov 03 18:14:51.491712 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:51.466586    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-workload-socket" pod "c3b8cf40-02df-499d-b41c-6e4bd9283a3b" (UID: "c3b8cf40-02df-499d-b41c-6e4bd9283a3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:51.491712 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:51.476495    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-istio-data" pod "c3b8cf40-02df-499d-b41c-6e4bd9283a3b" (UID: "c3b8cf40-02df-499d-b41c-6e4bd9283a3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:51.491712 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:51.476761    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-credential-socket" pod "c3b8cf40-02df-499d-b41c-6e4bd9283a3b" (UID: "c3b8cf40-02df-499d-b41c-6e4bd9283a3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:51.491712 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:51.477318    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-istiod-ca-cert" pod "c3b8cf40-02df-499d-b41c-6e4bd9283a3b" (UID: "c3b8cf40-02df-499d-b41c-6e4bd9283a3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:14:51.491712 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:51.477591    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-workload-certs" pod "c3b8cf40-02df-499d-b41c-6e4bd9283a3b" (UID: "c3b8cf40-02df-499d-b41c-6e4bd9283a3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:51.603650 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:51.514944    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-kube-api-access-hk2mm" pod "c3b8cf40-02df-499d-b41c-6e4bd9283a3b" (UID: "c3b8cf40-02df-499d-b41c-6e4bd9283a3b"). InnerVolumeSpecName "kube-api-access-hk2mm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:51.603650 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:51.562960    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:51.603650 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:51.562995    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:51.603650 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:51.563012    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk2mm\" (UniqueName: \"kubernetes.io/projected/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-kube-api-access-hk2mm\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:51.603650 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:51.563026    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:51.603650 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:51.563040    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:51.603650 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:51.563053    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:51.693926 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:51.693824    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-istio-token" pod "c3b8cf40-02df-499d-b41c-6e4bd9283a3b" (UID: "c3b8cf40-02df-499d-b41c-6e4bd9283a3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:51.871395 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:51.870869    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-istio-podinfo" pod "c3b8cf40-02df-499d-b41c-6e4bd9283a3b" (UID: "c3b8cf40-02df-499d-b41c-6e4bd9283a3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:14:51.872914 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:51.872864    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-istio-envoy" pod "c3b8cf40-02df-499d-b41c-6e4bd9283a3b" (UID: "c3b8cf40-02df-499d-b41c-6e4bd9283a3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:51.900681 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:51.900300    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69"
Nov 03 18:14:52.005577 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:51.953884    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:52.005577 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:51.953923    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:52.005577 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:51.953938    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3b8cf40-02df-499d-b41c-6e4bd9283a3b-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:52.005577 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:51.975765    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 03 18:14:52.005577 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:51.975825    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w" podUID="3d07e31d-20b5-47a1-901c-915565686286" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 03 18:14:52.039153 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:52.038942    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Nov 03 18:14:52.039427 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:52.039359    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg" podUID="daf3798f-da45-4287-94f6-eb32897e807b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Nov 03 18:14:52.072636 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:52.057081    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1bcb2fe-aef1-446a-9e93-a18dc824d388-istio-token\" (UniqueName: \"kubernetes.io/projected/a1bcb2fe-aef1-446a-9e93-a18dc824d388-istio-token\") pod \"a1bcb2fe-aef1-446a-9e93-a18dc824d388\" (UID: \"a1bcb2fe-aef1-446a-9e93-a18dc824d388\") "
Nov 03 18:14:52.257078 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:52.238770    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1bcb2fe-aef1-446a-9e93-a18dc824d388-kube-api-access-9c6r4\" (UniqueName: \"kubernetes.io/projected/a1bcb2fe-aef1-446a-9e93-a18dc824d388-kube-api-access-9c6r4\") pod \"a1bcb2fe-aef1-446a-9e93-a18dc824d388\" (UID: \"a1bcb2fe-aef1-446a-9e93-a18dc824d388\") "
Nov 03 18:14:52.257078 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:52.238822    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1bcb2fe-aef1-446a-9e93-a18dc824d388-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1bcb2fe-aef1-446a-9e93-a18dc824d388-workload-certs\") pod \"a1bcb2fe-aef1-446a-9e93-a18dc824d388\" (UID: \"a1bcb2fe-aef1-446a-9e93-a18dc824d388\") "
Nov 03 18:14:52.257078 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:52.238988    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1bcb2fe-aef1-446a-9e93-a18dc824d388-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bcb2fe-aef1-446a-9e93-a18dc824d388-credential-socket\") pod \"a1bcb2fe-aef1-446a-9e93-a18dc824d388\" (UID: \"a1bcb2fe-aef1-446a-9e93-a18dc824d388\") "
Nov 03 18:14:52.257078 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:52.239062    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1bcb2fe-aef1-446a-9e93-a18dc824d388-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bcb2fe-aef1-446a-9e93-a18dc824d388-workload-socket\") pod \"a1bcb2fe-aef1-446a-9e93-a18dc824d388\" (UID: \"a1bcb2fe-aef1-446a-9e93-a18dc824d388\") "
Nov 03 18:14:52.257078 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:52.239109    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1bcb2fe-aef1-446a-9e93-a18dc824d388-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1bcb2fe-aef1-446a-9e93-a18dc824d388-istio-data\") pod \"a1bcb2fe-aef1-446a-9e93-a18dc824d388\" (UID: \"a1bcb2fe-aef1-446a-9e93-a18dc824d388\") "
Nov 03 18:14:52.257078 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:52.239180    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1bcb2fe-aef1-446a-9e93-a18dc824d388-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1bcb2fe-aef1-446a-9e93-a18dc824d388-istiod-ca-cert\") pod \"a1bcb2fe-aef1-446a-9e93-a18dc824d388\" (UID: \"a1bcb2fe-aef1-446a-9e93-a18dc824d388\") "
Nov 03 18:14:52.257078 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:52.239232    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1bcb2fe-aef1-446a-9e93-a18dc824d388-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1bcb2fe-aef1-446a-9e93-a18dc824d388-istio-podinfo\") pod \"a1bcb2fe-aef1-446a-9e93-a18dc824d388\" (UID: \"a1bcb2fe-aef1-446a-9e93-a18dc824d388\") "
Nov 03 18:14:52.257078 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:52.239417    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1bcb2fe-aef1-446a-9e93-a18dc824d388-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1bcb2fe-aef1-446a-9e93-a18dc824d388-istio-envoy\") pod \"a1bcb2fe-aef1-446a-9e93-a18dc824d388\" (UID: \"a1bcb2fe-aef1-446a-9e93-a18dc824d388\") "
Nov 03 18:14:52.273633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:52.259711    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1bcb2fe-aef1-446a-9e93-a18dc824d388-workload-socket" pod "a1bcb2fe-aef1-446a-9e93-a18dc824d388" (UID: "a1bcb2fe-aef1-446a-9e93-a18dc824d388"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:52.273633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:52.266312    5315 generic.go:356] "Generic (PLEG): container finished" podID="1977c777-abc0-4d10-a0ee-3985dd2be5be" containerID="20c9b60fc479b95e3bf4ceb30bf0668e9fd0cb63001a00378e9709e988fe2cd1" exitCode=0
Nov 03 18:14:52.273633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:52.266487    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs" event={"ID":"1977c777-abc0-4d10-a0ee-3985dd2be5be","Type":"ContainerDied","Data":"20c9b60fc479b95e3bf4ceb30bf0668e9fd0cb63001a00378e9709e988fe2cd1"}
Nov 03 18:14:52.273633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:52.271817    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1bcb2fe-aef1-446a-9e93-a18dc824d388-istiod-ca-cert" pod "a1bcb2fe-aef1-446a-9e93-a18dc824d388" (UID: "a1bcb2fe-aef1-446a-9e93-a18dc824d388"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:14:52.318488 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:52.281851    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1bcb2fe-aef1-446a-9e93-a18dc824d388-istio-token" pod "a1bcb2fe-aef1-446a-9e93-a18dc824d388" (UID: "a1bcb2fe-aef1-446a-9e93-a18dc824d388"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:52.318488 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:52.293719    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1bcb2fe-aef1-446a-9e93-a18dc824d388-workload-certs" pod "a1bcb2fe-aef1-446a-9e93-a18dc824d388" (UID: "a1bcb2fe-aef1-446a-9e93-a18dc824d388"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:52.319256 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:52.319178    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1bcb2fe-aef1-446a-9e93-a18dc824d388-credential-socket" pod "a1bcb2fe-aef1-446a-9e93-a18dc824d388" (UID: "a1bcb2fe-aef1-446a-9e93-a18dc824d388"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:52.319733 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:52.319689    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1bcb2fe-aef1-446a-9e93-a18dc824d388-istio-data" pod "a1bcb2fe-aef1-446a-9e93-a18dc824d388" (UID: "a1bcb2fe-aef1-446a-9e93-a18dc824d388"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:52.374684 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:52.321952    5315 generic.go:356] "Generic (PLEG): container finished" podID="df589ab9-3107-4b55-9195-6ee315bd3965" containerID="1b8d55022722af293a4b45330a88ed9187b8c288ed428fc844fb82dca87afc04" exitCode=0
Nov 03 18:14:52.374684 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:52.322184    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m" event={"ID":"df589ab9-3107-4b55-9195-6ee315bd3965","Type":"ContainerDied","Data":"1b8d55022722af293a4b45330a88ed9187b8c288ed428fc844fb82dca87afc04"}
Nov 03 18:14:52.374684 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:52.338802    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1bcb2fe-aef1-446a-9e93-a18dc824d388-istio-envoy" pod "a1bcb2fe-aef1-446a-9e93-a18dc824d388" (UID: "a1bcb2fe-aef1-446a-9e93-a18dc824d388"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:52.374684 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:52.342967    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1bcb2fe-aef1-446a-9e93-a18dc824d388-kube-api-access-9c6r4" pod "a1bcb2fe-aef1-446a-9e93-a18dc824d388" (UID: "a1bcb2fe-aef1-446a-9e93-a18dc824d388"). InnerVolumeSpecName "kube-api-access-9c6r4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:52.374684 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:52.350713    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1bcb2fe-aef1-446a-9e93-a18dc824d388-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:52.374684 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:52.350747    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1bcb2fe-aef1-446a-9e93-a18dc824d388-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:52.374684 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:52.350780    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c6r4\" (UniqueName: \"kubernetes.io/projected/a1bcb2fe-aef1-446a-9e93-a18dc824d388-kube-api-access-9c6r4\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:52.374684 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:52.350796    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1bcb2fe-aef1-446a-9e93-a18dc824d388-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:52.374684 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:52.350810    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bcb2fe-aef1-446a-9e93-a18dc824d388-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:52.374684 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:52.350826    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bcb2fe-aef1-446a-9e93-a18dc824d388-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:52.374684 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:52.350841    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1bcb2fe-aef1-446a-9e93-a18dc824d388-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:52.374684 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:52.350856    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1bcb2fe-aef1-446a-9e93-a18dc824d388-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:52.377878 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:52.377820    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1bcb2fe-aef1-446a-9e93-a18dc824d388-istio-podinfo" pod "a1bcb2fe-aef1-446a-9e93-a18dc824d388" (UID: "a1bcb2fe-aef1-446a-9e93-a18dc824d388"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:14:52.399165 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:52.399019    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4" event={"ID":"c3b8cf40-02df-499d-b41c-6e4bd9283a3b","Type":"ContainerDied","Data":"4a864df2a434a7025ec9c065163ab233458937fc1c261d6c389988a83c3e2395"}
Nov 03 18:14:52.399639 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:52.399591    5315 scope.go:122] "RemoveContainer" containerID="94994b1046d894c179ec978ed84492241f9b524e40aaf36f07a96d48c0bbcf72"
Nov 03 18:14:52.400138 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:52.400093    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4"
Nov 03 18:14:52.421350 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:52.418950    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-1720c00c/e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr" podUID="8b7d3471-e6a0-4514-bf01-afb303808e53" containerName="main" probeResult="failure" output="Get \"https://10.133.0.167:8000/ready\": context deadline exceeded"
Nov 03 18:14:52.455162 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:52.455115    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1bcb2fe-aef1-446a-9e93-a18dc824d388-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:52.496993 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:52.483090    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69" event={"ID":"a1bcb2fe-aef1-446a-9e93-a18dc824d388","Type":"ContainerDied","Data":"f541bdcfdbc434a0b7e488ba8553e004de9a118ae409bf0b559818a521b95479"}
Nov 03 18:14:52.496993 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:52.483243    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69"
Nov 03 18:14:52.505862 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:52.505835    5315 scope.go:122] "RemoveContainer" containerID="0328b4a91e6c4a00505ad05fdbb246b65a375392a88f17f99f08423cef3b220c"
Nov 03 18:14:52.749932 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:52.745091    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt" podUID="7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:14:53.017673 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.010127    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 03 18:14:53.017673 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.010197    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w" podUID="3d07e31d-20b5-47a1-901c-915565686286" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 03 18:14:53.034635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.020261    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4"]
Nov 03 18:14:53.034635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.023694    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-g88m4"]
Nov 03 18:14:53.087296 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.087059    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Nov 03 18:14:53.087296 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.087118    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg" podUID="daf3798f-da45-4287-94f6-eb32897e807b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Nov 03 18:14:53.115421 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.115391    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs"
Nov 03 18:14:53.121748 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.121710    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3b8cf40-02df-499d-b41c-6e4bd9283a3b" path="/var/lib/kubelet/pods/c3b8cf40-02df-499d-b41c-6e4bd9283a3b/volumes"
Nov 03 18:14:53.122429 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.122407    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69"]
Nov 03 18:14:53.175480 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.172800    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-5wx69"]
Nov 03 18:14:53.182740 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.182702    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b01870c-v92xt"]
Nov 03 18:14:53.185522 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.185491    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b01870c-v92xt"]
Nov 03 18:14:53.221740 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.221713    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m"
Nov 03 18:14:53.312994 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.312957    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df589ab9-3107-4b55-9195-6ee315bd3965-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df589ab9-3107-4b55-9195-6ee315bd3965-istio-data\") pod \"df589ab9-3107-4b55-9195-6ee315bd3965\" (UID: \"df589ab9-3107-4b55-9195-6ee315bd3965\") "
Nov 03 18:14:53.313501 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.313032    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df589ab9-3107-4b55-9195-6ee315bd3965-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df589ab9-3107-4b55-9195-6ee315bd3965-istio-envoy\") pod \"df589ab9-3107-4b55-9195-6ee315bd3965\" (UID: \"df589ab9-3107-4b55-9195-6ee315bd3965\") "
Nov 03 18:14:53.313501 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.313069    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df589ab9-3107-4b55-9195-6ee315bd3965-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df589ab9-3107-4b55-9195-6ee315bd3965-istio-podinfo\") pod \"df589ab9-3107-4b55-9195-6ee315bd3965\" (UID: \"df589ab9-3107-4b55-9195-6ee315bd3965\") "
Nov 03 18:14:53.313501 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.313152    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df589ab9-3107-4b55-9195-6ee315bd3965-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df589ab9-3107-4b55-9195-6ee315bd3965-istiod-ca-cert\") pod \"df589ab9-3107-4b55-9195-6ee315bd3965\" (UID: \"df589ab9-3107-4b55-9195-6ee315bd3965\") "
Nov 03 18:14:53.313501 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.313201    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1977c777-abc0-4d10-a0ee-3985dd2be5be-kube-api-access-qzcrd\" (UniqueName: \"kubernetes.io/projected/1977c777-abc0-4d10-a0ee-3985dd2be5be-kube-api-access-qzcrd\") pod \"1977c777-abc0-4d10-a0ee-3985dd2be5be\" (UID: \"1977c777-abc0-4d10-a0ee-3985dd2be5be\") "
Nov 03 18:14:53.313501 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.313242    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1977c777-abc0-4d10-a0ee-3985dd2be5be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1977c777-abc0-4d10-a0ee-3985dd2be5be-workload-socket\") pod \"1977c777-abc0-4d10-a0ee-3985dd2be5be\" (UID: \"1977c777-abc0-4d10-a0ee-3985dd2be5be\") "
Nov 03 18:14:53.313501 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.313276    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df589ab9-3107-4b55-9195-6ee315bd3965-istio-data" pod "df589ab9-3107-4b55-9195-6ee315bd3965" (UID: "df589ab9-3107-4b55-9195-6ee315bd3965"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:53.313501 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.313299    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1977c777-abc0-4d10-a0ee-3985dd2be5be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1977c777-abc0-4d10-a0ee-3985dd2be5be-workload-certs\") pod \"1977c777-abc0-4d10-a0ee-3985dd2be5be\" (UID: \"1977c777-abc0-4d10-a0ee-3985dd2be5be\") "
Nov 03 18:14:53.313946 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.313564    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1977c777-abc0-4d10-a0ee-3985dd2be5be-workload-certs" pod "1977c777-abc0-4d10-a0ee-3985dd2be5be" (UID: "1977c777-abc0-4d10-a0ee-3985dd2be5be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:53.313946 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.313686    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df589ab9-3107-4b55-9195-6ee315bd3965-istiod-ca-cert" pod "df589ab9-3107-4b55-9195-6ee315bd3965" (UID: "df589ab9-3107-4b55-9195-6ee315bd3965"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:14:53.313946 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.313728    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df589ab9-3107-4b55-9195-6ee315bd3965-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df589ab9-3107-4b55-9195-6ee315bd3965-workload-certs\") pod \"df589ab9-3107-4b55-9195-6ee315bd3965\" (UID: \"df589ab9-3107-4b55-9195-6ee315bd3965\") "
Nov 03 18:14:53.313946 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.313815    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1977c777-abc0-4d10-a0ee-3985dd2be5be-workload-socket" pod "1977c777-abc0-4d10-a0ee-3985dd2be5be" (UID: "1977c777-abc0-4d10-a0ee-3985dd2be5be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:53.313946 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.313838    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df589ab9-3107-4b55-9195-6ee315bd3965-istio-token\" (UniqueName: \"kubernetes.io/projected/df589ab9-3107-4b55-9195-6ee315bd3965-istio-token\") pod \"df589ab9-3107-4b55-9195-6ee315bd3965\" (UID: \"df589ab9-3107-4b55-9195-6ee315bd3965\") "
Nov 03 18:14:53.313946 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.313874    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df589ab9-3107-4b55-9195-6ee315bd3965-kube-api-access-gkr29\" (UniqueName: \"kubernetes.io/projected/df589ab9-3107-4b55-9195-6ee315bd3965-kube-api-access-gkr29\") pod \"df589ab9-3107-4b55-9195-6ee315bd3965\" (UID: \"df589ab9-3107-4b55-9195-6ee315bd3965\") "
Nov 03 18:14:53.314273 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.313957    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df589ab9-3107-4b55-9195-6ee315bd3965-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df589ab9-3107-4b55-9195-6ee315bd3965-workload-socket\") pod \"df589ab9-3107-4b55-9195-6ee315bd3965\" (UID: \"df589ab9-3107-4b55-9195-6ee315bd3965\") "
Nov 03 18:14:53.314273 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.314024    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1977c777-abc0-4d10-a0ee-3985dd2be5be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1977c777-abc0-4d10-a0ee-3985dd2be5be-istio-podinfo\") pod \"1977c777-abc0-4d10-a0ee-3985dd2be5be\" (UID: \"1977c777-abc0-4d10-a0ee-3985dd2be5be\") "
Nov 03 18:14:53.314273 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.314074    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1977c777-abc0-4d10-a0ee-3985dd2be5be-istio-token\" (UniqueName: \"kubernetes.io/projected/1977c777-abc0-4d10-a0ee-3985dd2be5be-istio-token\") pod \"1977c777-abc0-4d10-a0ee-3985dd2be5be\" (UID: \"1977c777-abc0-4d10-a0ee-3985dd2be5be\") "
Nov 03 18:14:53.314273 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.314110    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1977c777-abc0-4d10-a0ee-3985dd2be5be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1977c777-abc0-4d10-a0ee-3985dd2be5be-istiod-ca-cert\") pod \"1977c777-abc0-4d10-a0ee-3985dd2be5be\" (UID: \"1977c777-abc0-4d10-a0ee-3985dd2be5be\") "
Nov 03 18:14:53.314273 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.314148    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df589ab9-3107-4b55-9195-6ee315bd3965-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df589ab9-3107-4b55-9195-6ee315bd3965-credential-socket\") pod \"df589ab9-3107-4b55-9195-6ee315bd3965\" (UID: \"df589ab9-3107-4b55-9195-6ee315bd3965\") "
Nov 03 18:14:53.314273 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.314207    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1977c777-abc0-4d10-a0ee-3985dd2be5be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1977c777-abc0-4d10-a0ee-3985dd2be5be-istio-envoy\") pod \"1977c777-abc0-4d10-a0ee-3985dd2be5be\" (UID: \"1977c777-abc0-4d10-a0ee-3985dd2be5be\") "
Nov 03 18:14:53.314273 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.314252    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1977c777-abc0-4d10-a0ee-3985dd2be5be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1977c777-abc0-4d10-a0ee-3985dd2be5be-istio-data\") pod \"1977c777-abc0-4d10-a0ee-3985dd2be5be\" (UID: \"1977c777-abc0-4d10-a0ee-3985dd2be5be\") "
Nov 03 18:14:53.314667 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.314288    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1977c777-abc0-4d10-a0ee-3985dd2be5be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1977c777-abc0-4d10-a0ee-3985dd2be5be-credential-socket\") pod \"1977c777-abc0-4d10-a0ee-3985dd2be5be\" (UID: \"1977c777-abc0-4d10-a0ee-3985dd2be5be\") "
Nov 03 18:14:53.314667 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.314349    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df589ab9-3107-4b55-9195-6ee315bd3965-workload-socket" pod "df589ab9-3107-4b55-9195-6ee315bd3965" (UID: "df589ab9-3107-4b55-9195-6ee315bd3965"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:53.315046 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.315010    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1977c777-abc0-4d10-a0ee-3985dd2be5be-istiod-ca-cert" pod "1977c777-abc0-4d10-a0ee-3985dd2be5be" (UID: "1977c777-abc0-4d10-a0ee-3985dd2be5be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:14:53.315274 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.315207    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1977c777-abc0-4d10-a0ee-3985dd2be5be-credential-socket" pod "1977c777-abc0-4d10-a0ee-3985dd2be5be" (UID: "1977c777-abc0-4d10-a0ee-3985dd2be5be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:53.315274 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.315140    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df589ab9-3107-4b55-9195-6ee315bd3965-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:53.315274 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.315255    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df589ab9-3107-4b55-9195-6ee315bd3965-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:53.315274 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.315272    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1977c777-abc0-4d10-a0ee-3985dd2be5be-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:53.315494 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.315286    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1977c777-abc0-4d10-a0ee-3985dd2be5be-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:53.315494 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.315299    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df589ab9-3107-4b55-9195-6ee315bd3965-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:53.315494 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.315313    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1977c777-abc0-4d10-a0ee-3985dd2be5be-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:53.315494 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.315403    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1977c777-abc0-4d10-a0ee-3985dd2be5be-istio-data" pod "1977c777-abc0-4d10-a0ee-3985dd2be5be" (UID: "1977c777-abc0-4d10-a0ee-3985dd2be5be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:53.315722 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.315526    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df589ab9-3107-4b55-9195-6ee315bd3965-credential-socket" pod "df589ab9-3107-4b55-9195-6ee315bd3965" (UID: "df589ab9-3107-4b55-9195-6ee315bd3965"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:53.315789 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.315750    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df589ab9-3107-4b55-9195-6ee315bd3965-workload-certs" pod "df589ab9-3107-4b55-9195-6ee315bd3965" (UID: "df589ab9-3107-4b55-9195-6ee315bd3965"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:53.317031 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.316997    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df589ab9-3107-4b55-9195-6ee315bd3965-istio-envoy" pod "df589ab9-3107-4b55-9195-6ee315bd3965" (UID: "df589ab9-3107-4b55-9195-6ee315bd3965"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:53.317820 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.317770    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df589ab9-3107-4b55-9195-6ee315bd3965-istio-token" pod "df589ab9-3107-4b55-9195-6ee315bd3965" (UID: "df589ab9-3107-4b55-9195-6ee315bd3965"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:53.318252 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.318175    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df589ab9-3107-4b55-9195-6ee315bd3965-istio-podinfo" pod "df589ab9-3107-4b55-9195-6ee315bd3965" (UID: "df589ab9-3107-4b55-9195-6ee315bd3965"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:14:53.318252 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.318192    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1977c777-abc0-4d10-a0ee-3985dd2be5be-istio-envoy" pod "1977c777-abc0-4d10-a0ee-3985dd2be5be" (UID: "1977c777-abc0-4d10-a0ee-3985dd2be5be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:53.318252 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.318206    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1977c777-abc0-4d10-a0ee-3985dd2be5be-istio-token" pod "1977c777-abc0-4d10-a0ee-3985dd2be5be" (UID: "1977c777-abc0-4d10-a0ee-3985dd2be5be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:53.318252 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.318234    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1977c777-abc0-4d10-a0ee-3985dd2be5be-kube-api-access-qzcrd" pod "1977c777-abc0-4d10-a0ee-3985dd2be5be" (UID: "1977c777-abc0-4d10-a0ee-3985dd2be5be"). InnerVolumeSpecName "kube-api-access-qzcrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:53.318580 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.318558    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1977c777-abc0-4d10-a0ee-3985dd2be5be-istio-podinfo" pod "1977c777-abc0-4d10-a0ee-3985dd2be5be" (UID: "1977c777-abc0-4d10-a0ee-3985dd2be5be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:14:53.319388 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.319360    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df589ab9-3107-4b55-9195-6ee315bd3965-kube-api-access-gkr29" pod "df589ab9-3107-4b55-9195-6ee315bd3965" (UID: "df589ab9-3107-4b55-9195-6ee315bd3965"). InnerVolumeSpecName "kube-api-access-gkr29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:53.419598 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.419561    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1977c777-abc0-4d10-a0ee-3985dd2be5be-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:53.419860 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.419839    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1977c777-abc0-4d10-a0ee-3985dd2be5be-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:53.419992 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.419978    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df589ab9-3107-4b55-9195-6ee315bd3965-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:53.420091 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.420078    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1977c777-abc0-4d10-a0ee-3985dd2be5be-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:53.420176 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.420163    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1977c777-abc0-4d10-a0ee-3985dd2be5be-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:53.420287 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.420272    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1977c777-abc0-4d10-a0ee-3985dd2be5be-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:53.420379 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.420367    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df589ab9-3107-4b55-9195-6ee315bd3965-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:53.420457 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.420446    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df589ab9-3107-4b55-9195-6ee315bd3965-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:53.420549 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.420535    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzcrd\" (UniqueName: \"kubernetes.io/projected/1977c777-abc0-4d10-a0ee-3985dd2be5be-kube-api-access-qzcrd\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:53.420677 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.420648    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df589ab9-3107-4b55-9195-6ee315bd3965-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:53.420780 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.420767    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df589ab9-3107-4b55-9195-6ee315bd3965-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:53.420869 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.420855    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkr29\" (UniqueName: \"kubernetes.io/projected/df589ab9-3107-4b55-9195-6ee315bd3965-kube-api-access-gkr29\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:53.507155 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.507116    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs" event={"ID":"1977c777-abc0-4d10-a0ee-3985dd2be5be","Type":"ContainerDied","Data":"c4a19d1b5b171de39fb44cf77c2eb9d57aea0d52f0ab8cbf8d241983f203c241"}
Nov 03 18:14:53.507370 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.507194    5315 scope.go:122] "RemoveContainer" containerID="20c9b60fc479b95e3bf4ceb30bf0668e9fd0cb63001a00378e9709e988fe2cd1"
Nov 03 18:14:53.507370 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.507130    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs"
Nov 03 18:14:53.509126 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.509100    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m" event={"ID":"df589ab9-3107-4b55-9195-6ee315bd3965","Type":"ContainerDied","Data":"ed094bb4ba311957cfff8b401e4da4a06623b700392c9e4c87fda3c07b164dc7"}
Nov 03 18:14:53.509234 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.509178    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m"
Nov 03 18:14:53.519946 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.519916    5315 scope.go:122] "RemoveContainer" containerID="1b8d55022722af293a4b45330a88ed9187b8c288ed428fc844fb82dca87afc04"
Nov 03 18:14:53.728635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.722911    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m"]
Nov 03 18:14:53.816668 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.811688    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m"]
Nov 03 18:14:53.904718 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.904630    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs"]
Nov 03 18:14:53.909749 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.909475    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-q7zjs"]
Nov 03 18:14:53.916298 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.916265    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-260dd0-g6x4q"]
Nov 03 18:14:53.919076 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.919049    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-260dd0-g6x4q"]
Nov 03 18:14:53.978737 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.978638    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 03 18:14:53.978737 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:53.978698    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w" podUID="3d07e31d-20b5-47a1-901c-915565686286" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 03 18:14:54.001256 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:54.001219    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Nov 03 18:14:54.001433 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:54.001281    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg" podUID="daf3798f-da45-4287-94f6-eb32897e807b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Nov 03 18:14:54.082294 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:54.082259    5315 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:14:54.082531 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:54.082507    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-dmg7m" podUID="df589ab9-3107-4b55-9195-6ee315bd3965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:14:54.605663 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:14:54.605625    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:14:54.949692 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:54.949014    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 03 18:14:54.949692 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:54.949082    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w" podUID="3d07e31d-20b5-47a1-901c-915565686286" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 03 18:14:55.000488 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:55.000451    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Nov 03 18:14:55.000672 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:55.000520    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg" podUID="daf3798f-da45-4287-94f6-eb32897e807b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Nov 03 18:14:55.106253 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:55.106222    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1977c777-abc0-4d10-a0ee-3985dd2be5be" path="/var/lib/kubelet/pods/1977c777-abc0-4d10-a0ee-3985dd2be5be/volumes"
Nov 03 18:14:55.106934 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:55.106917    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81f2c5ce-15fa-4010-b323-5cddd39d5667" path="/var/lib/kubelet/pods/81f2c5ce-15fa-4010-b323-5cddd39d5667/volumes"
Nov 03 18:14:55.107415 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:55.107401    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9322d0f4-e8ca-483a-894e-62491681581c" path="/var/lib/kubelet/pods/9322d0f4-e8ca-483a-894e-62491681581c/volumes"
Nov 03 18:14:55.108368 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:55.108352    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1bcb2fe-aef1-446a-9e93-a18dc824d388" path="/var/lib/kubelet/pods/a1bcb2fe-aef1-446a-9e93-a18dc824d388/volumes"
Nov 03 18:14:55.108915 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:55.108901    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df589ab9-3107-4b55-9195-6ee315bd3965" path="/var/lib/kubelet/pods/df589ab9-3107-4b55-9195-6ee315bd3965/volumes"
Nov 03 18:14:55.907786 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:55.907739    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb"]
Nov 03 18:14:55.908325 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:55.907990    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb" podUID="c4428ff6-1929-4fb0-82ee-ab6a785ca9e5" containerName="istio-proxy" containerID="cri-o://459e5409eec81d918c9469ae21ec9004f857a88c259bf3ff5bf625f9f3760b59" gracePeriod=30
Nov 03 18:14:55.912738 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:55.912699    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv"]
Nov 03 18:14:55.913402 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:55.912923    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv" podUID="a36c9b31-8ab0-449f-8fb4-87a702ab7abf" containerName="istio-proxy" containerID="cri-o://416a1054372c9b857a182516290a16c306c50e7e7da97e7c03e19d464ce85f71" gracePeriod=30
Nov 03 18:14:55.918510 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:55.918482    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl"]
Nov 03 18:14:55.918738 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:55.918713    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl" podUID="ede64d2a-75d4-4d21-8592-1680fb138e4e" containerName="istio-proxy" containerID="cri-o://5d0e5bd47c52ec88bb63cf23f78248dfef2112f33d18d424fce6bb5bead52493" gracePeriod=30
Nov 03 18:14:55.947114 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:55.947061    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 03 18:14:55.947329 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:55.947130    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w" podUID="3d07e31d-20b5-47a1-901c-915565686286" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 03 18:14:56.004669 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:56.002575    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Nov 03 18:14:56.004669 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:56.002657    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg" podUID="daf3798f-da45-4287-94f6-eb32897e807b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Nov 03 18:14:57.525260 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:57.525213    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w"
Nov 03 18:14:57.528151 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:57.526035    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w"
Nov 03 18:14:57.528492 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:57.528466    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w"
Nov 03 18:14:57.935632 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:57.933154    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:14:57.935632 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:57.933236    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv" podUID="a36c9b31-8ab0-449f-8fb4-87a702ab7abf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:14:58.002196 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:58.002146    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:14:58.002396 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:58.002230    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg" podUID="daf3798f-da45-4287-94f6-eb32897e807b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:14:58.149630 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:58.149308    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:14:58.149630 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:58.149397    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb" podUID="c4428ff6-1929-4fb0-82ee-ab6a785ca9e5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:14:58.335945 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:58.334841    5315 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:14:58.335945 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:58.334920    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl" podUID="ede64d2a-75d4-4d21-8592-1680fb138e4e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:14:59.015089 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:59.015044    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:14:59.015478 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:59.015119    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg" podUID="daf3798f-da45-4287-94f6-eb32897e807b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:14:59.029478 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:59.025782    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg"
Nov 03 18:14:59.029478 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:59.027465    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg"
Nov 03 18:14:59.032180 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:14:59.030231    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg"
Nov 03 18:15:00.165526 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.165485    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895495-r68gb"]
Nov 03 18:15:00.166196 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.166177    5315 cpu_manager.go:414] "containerMap: removing container" podUID="a1bcb2fe-aef1-446a-9e93-a18dc824d388" containerName="istio-proxy"
Nov 03 18:15:00.166196 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.166196    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1bcb2fe-aef1-446a-9e93-a18dc824d388" containerName="istio-proxy"
Nov 03 18:15:00.166381 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.166251    5315 cpu_manager.go:414] "containerMap: removing container" podUID="df589ab9-3107-4b55-9195-6ee315bd3965" containerName="istio-proxy"
Nov 03 18:15:00.166381 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.166260    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df589ab9-3107-4b55-9195-6ee315bd3965" containerName="istio-proxy"
Nov 03 18:15:00.166381 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.166274    5315 cpu_manager.go:414] "containerMap: removing container" podUID="c3b8cf40-02df-499d-b41c-6e4bd9283a3b" containerName="istio-proxy"
Nov 03 18:15:00.166381 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.166283    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3b8cf40-02df-499d-b41c-6e4bd9283a3b" containerName="istio-proxy"
Nov 03 18:15:00.166381 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.166306    5315 cpu_manager.go:414] "containerMap: removing container" podUID="1977c777-abc0-4d10-a0ee-3985dd2be5be" containerName="istio-proxy"
Nov 03 18:15:00.166381 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.166312    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1977c777-abc0-4d10-a0ee-3985dd2be5be" containerName="istio-proxy"
Nov 03 18:15:00.166678 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.166509    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="df589ab9-3107-4b55-9195-6ee315bd3965" containerName="istio-proxy"
Nov 03 18:15:00.166678 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.166527    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3b8cf40-02df-499d-b41c-6e4bd9283a3b" containerName="istio-proxy"
Nov 03 18:15:00.166678 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.166541    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="1977c777-abc0-4d10-a0ee-3985dd2be5be" containerName="istio-proxy"
Nov 03 18:15:00.166678 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.166555    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1bcb2fe-aef1-446a-9e93-a18dc824d388" containerName="istio-proxy"
Nov 03 18:15:00.172762 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.172740    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895495-jgzv4"]
Nov 03 18:15:00.172945 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.172926    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895495-r68gb"
Nov 03 18:15:00.177469 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.177446    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c1b2d2-29895495-k5x2m"]
Nov 03 18:15:00.177591 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.177538    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-jgzv4"
Nov 03 18:15:00.182381 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.182356    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c1b2d2-29895495-k5x2m"
Nov 03 18:15:00.212897 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.212070    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895495-shccb"]
Nov 03 18:15:00.215939 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.215900    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9r5l\" (UniqueName: \"kubernetes.io/projected/5d04a4ae-44e9-4f14-ada5-855c1efe7539-kube-api-access-v9r5l\") pod \"maas-api-key-cleanup-e2e-worker-w4-a86228-29895495-r68gb\" (UID: \"5d04a4ae-44e9-4f14-ada5-855c1efe7539\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895495-r68gb"
Nov 03 18:15:00.216223 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.216201    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d04a4ae-44e9-4f14-ada5-855c1efe7539-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-a86228-29895495-r68gb\" (UID: \"5d04a4ae-44e9-4f14-ada5-855c1efe7539\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895495-r68gb"
Nov 03 18:15:00.216931 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.216912    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7a7c76-29895495-lhj7q"]
Nov 03 18:15:00.217048 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.217031    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895495-shccb"
Nov 03 18:15:00.221523 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.221503    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-80aca1-29895495-qvtbt"]
Nov 03 18:15:00.221664 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.221648    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7a7c76-29895495-lhj7q"
Nov 03 18:15:00.224243 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.224214    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:15:00.224426 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.224406    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:15:00.230675 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.228876    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-80aca1-29895495-qvtbt"
Nov 03 18:15:00.254385 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.254359    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:15:00.272775 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.272728    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895495-r68gb"]
Nov 03 18:15:00.278308 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.278276    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895495-jgzv4"]
Nov 03 18:15:00.283420 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.283389    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c1b2d2-29895495-k5x2m"]
Nov 03 18:15:00.317029 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.316990    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9r5l\" (UniqueName: \"kubernetes.io/projected/5d04a4ae-44e9-4f14-ada5-855c1efe7539-kube-api-access-v9r5l\") pod \"maas-api-key-cleanup-e2e-worker-w4-a86228-29895495-r68gb\" (UID: \"5d04a4ae-44e9-4f14-ada5-855c1efe7539\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895495-r68gb"
Nov 03 18:15:00.317179 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.317053    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj8pw\" (UniqueName: \"kubernetes.io/projected/8d8504d9-e3e9-4542-8e76-3dbd404ed89f-kube-api-access-gj8pw\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-7a7c76-29895495-lhj7q\" (UID: \"8d8504d9-e3e9-4542-8e76-3dbd404ed89f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7a7c76-29895495-lhj7q"
Nov 03 18:15:00.317179 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.317156    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54bd478e-0685-4ec9-8aa0-5c3bd73c56c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-f79c02-29895495-shccb\" (UID: \"54bd478e-0685-4ec9-8aa0-5c3bd73c56c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895495-shccb"
Nov 03 18:15:00.317278 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.317209    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26d8c8d7-89c0-47af-b90c-8b0677f71886-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-80aca1-29895495-qvtbt\" (UID: \"26d8c8d7-89c0-47af-b90c-8b0677f71886\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-80aca1-29895495-qvtbt"
Nov 03 18:15:00.317278 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.317239    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s7tp\" (UniqueName: \"kubernetes.io/projected/e31a2c86-cff2-49ed-8f40-4020dfef4808-kube-api-access-5s7tp\") pod \"maas-api-key-cleanup-29895495-jgzv4\" (UID: \"e31a2c86-cff2-49ed-8f40-4020dfef4808\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-jgzv4"
Nov 03 18:15:00.317404 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.317382    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e31a2c86-cff2-49ed-8f40-4020dfef4808-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895495-jgzv4\" (UID: \"e31a2c86-cff2-49ed-8f40-4020dfef4808\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-jgzv4"
Nov 03 18:15:00.317454 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.317419    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d04a4ae-44e9-4f14-ada5-855c1efe7539-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-a86228-29895495-r68gb\" (UID: \"5d04a4ae-44e9-4f14-ada5-855c1efe7539\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895495-r68gb"
Nov 03 18:15:00.317454 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.317443    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbvzh\" (UniqueName: \"kubernetes.io/projected/e3a2676b-0fd6-4c39-9ed7-d468706aa9e9-kube-api-access-gbvzh\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c1b2d2-29895495-k5x2m\" (UID: \"e3a2676b-0fd6-4c39-9ed7-d468706aa9e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c1b2d2-29895495-k5x2m"
Nov 03 18:15:00.317561 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.317485    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sswlc\" (UniqueName: \"kubernetes.io/projected/54bd478e-0685-4ec9-8aa0-5c3bd73c56c2-kube-api-access-sswlc\") pod \"maas-api-key-cleanup-e2e-worker-w2-f79c02-29895495-shccb\" (UID: \"54bd478e-0685-4ec9-8aa0-5c3bd73c56c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895495-shccb"
Nov 03 18:15:00.317630 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.317595    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpw8j\" (UniqueName: \"kubernetes.io/projected/26d8c8d7-89c0-47af-b90c-8b0677f71886-kube-api-access-kpw8j\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-80aca1-29895495-qvtbt\" (UID: \"26d8c8d7-89c0-47af-b90c-8b0677f71886\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-80aca1-29895495-qvtbt"
Nov 03 18:15:00.317687 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.317661    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3a2676b-0fd6-4c39-9ed7-d468706aa9e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c1b2d2-29895495-k5x2m\" (UID: \"e3a2676b-0fd6-4c39-9ed7-d468706aa9e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c1b2d2-29895495-k5x2m"
Nov 03 18:15:00.317752 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.317717    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d8504d9-e3e9-4542-8e76-3dbd404ed89f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-7a7c76-29895495-lhj7q\" (UID: \"8d8504d9-e3e9-4542-8e76-3dbd404ed89f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7a7c76-29895495-lhj7q"
Nov 03 18:15:00.318579 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.318545    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d04a4ae-44e9-4f14-ada5-855c1efe7539-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-a86228-29895495-r68gb\" (UID: \"5d04a4ae-44e9-4f14-ada5-855c1efe7539\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895495-r68gb"
Nov 03 18:15:00.325723 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.325692    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895495-shccb"]
Nov 03 18:15:00.331595 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.331225    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-80aca1-29895495-qvtbt"]
Nov 03 18:15:00.334352 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.334324    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7a7c76-29895495-lhj7q"]
Nov 03 18:15:00.375047 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.375019    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9r5l\" (UniqueName: \"kubernetes.io/projected/5d04a4ae-44e9-4f14-ada5-855c1efe7539-kube-api-access-v9r5l\") pod \"maas-api-key-cleanup-e2e-worker-w4-a86228-29895495-r68gb\" (UID: \"5d04a4ae-44e9-4f14-ada5-855c1efe7539\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895495-r68gb"
Nov 03 18:15:00.418960 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.418872    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj8pw\" (UniqueName: \"kubernetes.io/projected/8d8504d9-e3e9-4542-8e76-3dbd404ed89f-kube-api-access-gj8pw\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-7a7c76-29895495-lhj7q\" (UID: \"8d8504d9-e3e9-4542-8e76-3dbd404ed89f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7a7c76-29895495-lhj7q"
Nov 03 18:15:00.418960 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.418923    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54bd478e-0685-4ec9-8aa0-5c3bd73c56c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-f79c02-29895495-shccb\" (UID: \"54bd478e-0685-4ec9-8aa0-5c3bd73c56c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895495-shccb"
Nov 03 18:15:00.418960 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.418961    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26d8c8d7-89c0-47af-b90c-8b0677f71886-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-80aca1-29895495-qvtbt\" (UID: \"26d8c8d7-89c0-47af-b90c-8b0677f71886\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-80aca1-29895495-qvtbt"
Nov 03 18:15:00.419244 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.418985    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s7tp\" (UniqueName: \"kubernetes.io/projected/e31a2c86-cff2-49ed-8f40-4020dfef4808-kube-api-access-5s7tp\") pod \"maas-api-key-cleanup-29895495-jgzv4\" (UID: \"e31a2c86-cff2-49ed-8f40-4020dfef4808\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-jgzv4"
Nov 03 18:15:00.419244 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.419027    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e31a2c86-cff2-49ed-8f40-4020dfef4808-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895495-jgzv4\" (UID: \"e31a2c86-cff2-49ed-8f40-4020dfef4808\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-jgzv4"
Nov 03 18:15:00.419244 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.419060    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbvzh\" (UniqueName: \"kubernetes.io/projected/e3a2676b-0fd6-4c39-9ed7-d468706aa9e9-kube-api-access-gbvzh\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c1b2d2-29895495-k5x2m\" (UID: \"e3a2676b-0fd6-4c39-9ed7-d468706aa9e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c1b2d2-29895495-k5x2m"
Nov 03 18:15:00.419244 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.419093    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sswlc\" (UniqueName: \"kubernetes.io/projected/54bd478e-0685-4ec9-8aa0-5c3bd73c56c2-kube-api-access-sswlc\") pod \"maas-api-key-cleanup-e2e-worker-w2-f79c02-29895495-shccb\" (UID: \"54bd478e-0685-4ec9-8aa0-5c3bd73c56c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895495-shccb"
Nov 03 18:15:00.419244 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.419161    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpw8j\" (UniqueName: \"kubernetes.io/projected/26d8c8d7-89c0-47af-b90c-8b0677f71886-kube-api-access-kpw8j\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-80aca1-29895495-qvtbt\" (UID: \"26d8c8d7-89c0-47af-b90c-8b0677f71886\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-80aca1-29895495-qvtbt"
Nov 03 18:15:00.419244 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.419194    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3a2676b-0fd6-4c39-9ed7-d468706aa9e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c1b2d2-29895495-k5x2m\" (UID: \"e3a2676b-0fd6-4c39-9ed7-d468706aa9e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c1b2d2-29895495-k5x2m"
Nov 03 18:15:00.419585 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.419275    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d8504d9-e3e9-4542-8e76-3dbd404ed89f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-7a7c76-29895495-lhj7q\" (UID: \"8d8504d9-e3e9-4542-8e76-3dbd404ed89f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7a7c76-29895495-lhj7q"
Nov 03 18:15:00.419817 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.419793    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26d8c8d7-89c0-47af-b90c-8b0677f71886-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-80aca1-29895495-qvtbt\" (UID: \"26d8c8d7-89c0-47af-b90c-8b0677f71886\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-80aca1-29895495-qvtbt"
Nov 03 18:15:00.419894 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.419863    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54bd478e-0685-4ec9-8aa0-5c3bd73c56c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-f79c02-29895495-shccb\" (UID: \"54bd478e-0685-4ec9-8aa0-5c3bd73c56c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895495-shccb"
Nov 03 18:15:00.419947 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.419906    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e31a2c86-cff2-49ed-8f40-4020dfef4808-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895495-jgzv4\" (UID: \"e31a2c86-cff2-49ed-8f40-4020dfef4808\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-jgzv4"
Nov 03 18:15:00.420030 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.420008    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d8504d9-e3e9-4542-8e76-3dbd404ed89f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-7a7c76-29895495-lhj7q\" (UID: \"8d8504d9-e3e9-4542-8e76-3dbd404ed89f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7a7c76-29895495-lhj7q"
Nov 03 18:15:00.420168 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.420143    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3a2676b-0fd6-4c39-9ed7-d468706aa9e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c1b2d2-29895495-k5x2m\" (UID: \"e3a2676b-0fd6-4c39-9ed7-d468706aa9e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c1b2d2-29895495-k5x2m"
Nov 03 18:15:00.476850 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.476811    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpw8j\" (UniqueName: \"kubernetes.io/projected/26d8c8d7-89c0-47af-b90c-8b0677f71886-kube-api-access-kpw8j\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-80aca1-29895495-qvtbt\" (UID: \"26d8c8d7-89c0-47af-b90c-8b0677f71886\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-80aca1-29895495-qvtbt"
Nov 03 18:15:00.476850 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.476836    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj8pw\" (UniqueName: \"kubernetes.io/projected/8d8504d9-e3e9-4542-8e76-3dbd404ed89f-kube-api-access-gj8pw\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-7a7c76-29895495-lhj7q\" (UID: \"8d8504d9-e3e9-4542-8e76-3dbd404ed89f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7a7c76-29895495-lhj7q"
Nov 03 18:15:00.477060 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.476812    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sswlc\" (UniqueName: \"kubernetes.io/projected/54bd478e-0685-4ec9-8aa0-5c3bd73c56c2-kube-api-access-sswlc\") pod \"maas-api-key-cleanup-e2e-worker-w2-f79c02-29895495-shccb\" (UID: \"54bd478e-0685-4ec9-8aa0-5c3bd73c56c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895495-shccb"
Nov 03 18:15:00.477060 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.476925    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbvzh\" (UniqueName: \"kubernetes.io/projected/e3a2676b-0fd6-4c39-9ed7-d468706aa9e9-kube-api-access-gbvzh\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c1b2d2-29895495-k5x2m\" (UID: \"e3a2676b-0fd6-4c39-9ed7-d468706aa9e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c1b2d2-29895495-k5x2m"
Nov 03 18:15:00.477060 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.476970    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s7tp\" (UniqueName: \"kubernetes.io/projected/e31a2c86-cff2-49ed-8f40-4020dfef4808-kube-api-access-5s7tp\") pod \"maas-api-key-cleanup-29895495-jgzv4\" (UID: \"e31a2c86-cff2-49ed-8f40-4020dfef4808\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-jgzv4"
Nov 03 18:15:00.489603 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.489574    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895495-r68gb"
Nov 03 18:15:00.499231 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.499079    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-jgzv4"
Nov 03 18:15:00.504421 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.504397    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c1b2d2-29895495-k5x2m"
Nov 03 18:15:00.542249 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.541025    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895495-shccb"
Nov 03 18:15:00.554914 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.551231    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7a7c76-29895495-lhj7q"
Nov 03 18:15:00.556985 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.556886    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-80aca1-29895495-qvtbt"
Nov 03 18:15:00.918275 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.918243    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c1b2d2-29895495-k5x2m"]
Nov 03 18:15:00.922412 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:15:00.922370    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3a2676b_0fd6_4c39_9ed7_d468706aa9e9.slice/crio-2ddaba5be9bd7d5a5eb0006478482141e992be06542ec1bfd35b1d975bdec356 WatchSource:0}: Error finding container 2ddaba5be9bd7d5a5eb0006478482141e992be06542ec1bfd35b1d975bdec356: Status 404 returned error can't find the container with id 2ddaba5be9bd7d5a5eb0006478482141e992be06542ec1bfd35b1d975bdec356
Nov 03 18:15:00.977202 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.977167    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d"]
Nov 03 18:15:00.977628 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.977565    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d" podUID="ae6e0c1a-a26e-4b44-8e94-540944db8bd8" containerName="istio-proxy" containerID="cri-o://76fc11ded9455c2a81036b73e1425af8d992264ffeb97286ccc3e0acb07b726c" gracePeriod=30
Nov 03 18:15:00.996491 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:00.996462    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895495-shccb"]
Nov 03 18:15:01.019700 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.019671    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7a7c76-29895495-lhj7q"]
Nov 03 18:15:01.023691 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:15:01.022639    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54bd478e_0685_4ec9_8aa0_5c3bd73c56c2.slice/crio-0109f14428fdbd14b59ecb232e0db778204832840eb291ef6732e901498b9724 WatchSource:0}: Error finding container 0109f14428fdbd14b59ecb232e0db778204832840eb291ef6732e901498b9724: Status 404 returned error can't find the container with id 0109f14428fdbd14b59ecb232e0db778204832840eb291ef6732e901498b9724
Nov 03 18:15:01.029550 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.029520    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-80aca1-29895495-qvtbt"]
Nov 03 18:15:01.053486 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:15:01.053448    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26d8c8d7_89c0_47af_b90c_8b0677f71886.slice/crio-2a82be8d5e7829525d6503f8761e03aedd501b5734e44b700d48c899b85f0532 WatchSource:0}: Error finding container 2a82be8d5e7829525d6503f8761e03aedd501b5734e44b700d48c899b85f0532: Status 404 returned error can't find the container with id 2a82be8d5e7829525d6503f8761e03aedd501b5734e44b700d48c899b85f0532
Nov 03 18:15:01.311230 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.311180    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895495-r68gb"]
Nov 03 18:15:01.318026 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.317844    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895495-jgzv4"]
Nov 03 18:15:01.542853 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.542824    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb"
Nov 03 18:15:01.556714 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.556684    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl"
Nov 03 18:15:01.560452 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.560426    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv"
Nov 03 18:15:01.646107 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.646067    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-jgzv4" event={"ID":"e31a2c86-cff2-49ed-8f40-4020dfef4808","Type":"ContainerStarted","Data":"428006d9de17c40e2114647c1f3d3707dd56b62c47fe4cb0828689db0665e1a4"}
Nov 03 18:15:01.647478 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.647450    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895495-r68gb" event={"ID":"5d04a4ae-44e9-4f14-ada5-855c1efe7539","Type":"ContainerStarted","Data":"4c5e8935819a297c6c0ec12562cb7099d66e968e0e5b0e5c8edc4b30042638eb"}
Nov 03 18:15:01.648847 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.648822    5315 generic.go:356] "Generic (PLEG): container finished" podID="ede64d2a-75d4-4d21-8592-1680fb138e4e" containerID="5d0e5bd47c52ec88bb63cf23f78248dfef2112f33d18d424fce6bb5bead52493" exitCode=0
Nov 03 18:15:01.648965 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.648871    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl"
Nov 03 18:15:01.648965 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.648907    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl" event={"ID":"ede64d2a-75d4-4d21-8592-1680fb138e4e","Type":"ContainerDied","Data":"5d0e5bd47c52ec88bb63cf23f78248dfef2112f33d18d424fce6bb5bead52493"}
Nov 03 18:15:01.649077 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.648968    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl" event={"ID":"ede64d2a-75d4-4d21-8592-1680fb138e4e","Type":"ContainerDied","Data":"b6a5444be3aaf0ebdb403638fc42ee12578aa1cf55735a39e5007afedb4f43a5"}
Nov 03 18:15:01.649077 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.649000    5315 scope.go:122] "RemoveContainer" containerID="5d0e5bd47c52ec88bb63cf23f78248dfef2112f33d18d424fce6bb5bead52493"
Nov 03 18:15:01.650316 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.650290    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7a7c76-29895495-lhj7q" event={"ID":"8d8504d9-e3e9-4542-8e76-3dbd404ed89f","Type":"ContainerStarted","Data":"940909dfd9d4e115fe39124c3a5343c01d31b75c0a1ddf32981ddbe3ecefdcd2"}
Nov 03 18:15:01.651653 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.651631    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895495-shccb" event={"ID":"54bd478e-0685-4ec9-8aa0-5c3bd73c56c2","Type":"ContainerStarted","Data":"0109f14428fdbd14b59ecb232e0db778204832840eb291ef6732e901498b9724"}
Nov 03 18:15:01.653392 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.653368    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c1b2d2-29895495-k5x2m" event={"ID":"e3a2676b-0fd6-4c39-9ed7-d468706aa9e9","Type":"ContainerStarted","Data":"8a3016b2945d5b575495d0ddcb52cbd7ce892b95f35ff2f011684eb1c229e65e"}
Nov 03 18:15:01.653484 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.653414    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c1b2d2-29895495-k5x2m" event={"ID":"e3a2676b-0fd6-4c39-9ed7-d468706aa9e9","Type":"ContainerStarted","Data":"2ddaba5be9bd7d5a5eb0006478482141e992be06542ec1bfd35b1d975bdec356"}
Nov 03 18:15:01.654893 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.654871    5315 generic.go:356] "Generic (PLEG): container finished" podID="c4428ff6-1929-4fb0-82ee-ab6a785ca9e5" containerID="459e5409eec81d918c9469ae21ec9004f857a88c259bf3ff5bf625f9f3760b59" exitCode=0
Nov 03 18:15:01.654996 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.654923    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb"
Nov 03 18:15:01.654996 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.654948    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb" event={"ID":"c4428ff6-1929-4fb0-82ee-ab6a785ca9e5","Type":"ContainerDied","Data":"459e5409eec81d918c9469ae21ec9004f857a88c259bf3ff5bf625f9f3760b59"}
Nov 03 18:15:01.654996 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.654994    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb" event={"ID":"c4428ff6-1929-4fb0-82ee-ab6a785ca9e5","Type":"ContainerDied","Data":"e6719d16c14c7ffbb704b2b57bbcd46a567575fb2389e4ab9eceb8aa6c424380"}
Nov 03 18:15:01.656228 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.656208    5315 generic.go:356] "Generic (PLEG): container finished" podID="a36c9b31-8ab0-449f-8fb4-87a702ab7abf" containerID="416a1054372c9b857a182516290a16c306c50e7e7da97e7c03e19d464ce85f71" exitCode=0
Nov 03 18:15:01.656342 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.656248    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv"
Nov 03 18:15:01.656342 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.656278    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv" event={"ID":"a36c9b31-8ab0-449f-8fb4-87a702ab7abf","Type":"ContainerDied","Data":"416a1054372c9b857a182516290a16c306c50e7e7da97e7c03e19d464ce85f71"}
Nov 03 18:15:01.656342 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.656316    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv" event={"ID":"a36c9b31-8ab0-449f-8fb4-87a702ab7abf","Type":"ContainerDied","Data":"60d1b9be4df4c908923dedfc2424f28b6384e35ab6177b137558b23dfcb4b47e"}
Nov 03 18:15:01.657423 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.657400    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-80aca1-29895495-qvtbt" event={"ID":"26d8c8d7-89c0-47af-b90c-8b0677f71886","Type":"ContainerStarted","Data":"2a82be8d5e7829525d6503f8761e03aedd501b5734e44b700d48c899b85f0532"}
Nov 03 18:15:01.661160 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.661138    5315 scope.go:122] "RemoveContainer" containerID="5d0e5bd47c52ec88bb63cf23f78248dfef2112f33d18d424fce6bb5bead52493"
Nov 03 18:15:01.661482 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:15:01.661458    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d0e5bd47c52ec88bb63cf23f78248dfef2112f33d18d424fce6bb5bead52493\": container with ID starting with 5d0e5bd47c52ec88bb63cf23f78248dfef2112f33d18d424fce6bb5bead52493 not found: ID does not exist" containerID="5d0e5bd47c52ec88bb63cf23f78248dfef2112f33d18d424fce6bb5bead52493"
Nov 03 18:15:01.661547 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.661492    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d0e5bd47c52ec88bb63cf23f78248dfef2112f33d18d424fce6bb5bead52493"} err="failed to get container status \"5d0e5bd47c52ec88bb63cf23f78248dfef2112f33d18d424fce6bb5bead52493\": rpc error: code = NotFound desc = could not find container \"5d0e5bd47c52ec88bb63cf23f78248dfef2112f33d18d424fce6bb5bead52493\": container with ID starting with 5d0e5bd47c52ec88bb63cf23f78248dfef2112f33d18d424fce6bb5bead52493 not found: ID does not exist"
Nov 03 18:15:01.661547 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.661509    5315 scope.go:122] "RemoveContainer" containerID="459e5409eec81d918c9469ae21ec9004f857a88c259bf3ff5bf625f9f3760b59"
Nov 03 18:15:01.673912 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.673848    5315 scope.go:122] "RemoveContainer" containerID="459e5409eec81d918c9469ae21ec9004f857a88c259bf3ff5bf625f9f3760b59"
Nov 03 18:15:01.674323 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:15:01.674287    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"459e5409eec81d918c9469ae21ec9004f857a88c259bf3ff5bf625f9f3760b59\": container with ID starting with 459e5409eec81d918c9469ae21ec9004f857a88c259bf3ff5bf625f9f3760b59 not found: ID does not exist" containerID="459e5409eec81d918c9469ae21ec9004f857a88c259bf3ff5bf625f9f3760b59"
Nov 03 18:15:01.674405 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.674335    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"459e5409eec81d918c9469ae21ec9004f857a88c259bf3ff5bf625f9f3760b59"} err="failed to get container status \"459e5409eec81d918c9469ae21ec9004f857a88c259bf3ff5bf625f9f3760b59\": rpc error: code = NotFound desc = could not find container \"459e5409eec81d918c9469ae21ec9004f857a88c259bf3ff5bf625f9f3760b59\": container with ID starting with 459e5409eec81d918c9469ae21ec9004f857a88c259bf3ff5bf625f9f3760b59 not found: ID does not exist"
Nov 03 18:15:01.674405 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.674355    5315 scope.go:122] "RemoveContainer" containerID="416a1054372c9b857a182516290a16c306c50e7e7da97e7c03e19d464ce85f71"
Nov 03 18:15:01.689421 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.689392    5315 scope.go:122] "RemoveContainer" containerID="416a1054372c9b857a182516290a16c306c50e7e7da97e7c03e19d464ce85f71"
Nov 03 18:15:01.689840 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:15:01.689814    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"416a1054372c9b857a182516290a16c306c50e7e7da97e7c03e19d464ce85f71\": container with ID starting with 416a1054372c9b857a182516290a16c306c50e7e7da97e7c03e19d464ce85f71 not found: ID does not exist" containerID="416a1054372c9b857a182516290a16c306c50e7e7da97e7c03e19d464ce85f71"
Nov 03 18:15:01.689956 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.689849    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"416a1054372c9b857a182516290a16c306c50e7e7da97e7c03e19d464ce85f71"} err="failed to get container status \"416a1054372c9b857a182516290a16c306c50e7e7da97e7c03e19d464ce85f71\": rpc error: code = NotFound desc = could not find container \"416a1054372c9b857a182516290a16c306c50e7e7da97e7c03e19d464ce85f71\": container with ID starting with 416a1054372c9b857a182516290a16c306c50e7e7da97e7c03e19d464ce85f71 not found: ID does not exist"
Nov 03 18:15:01.703433 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.703397    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede64d2a-75d4-4d21-8592-1680fb138e4e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ede64d2a-75d4-4d21-8592-1680fb138e4e-istio-envoy\") pod \"ede64d2a-75d4-4d21-8592-1680fb138e4e\" (UID: \"ede64d2a-75d4-4d21-8592-1680fb138e4e\") "
Nov 03 18:15:01.704484 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.704132    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-istio-podinfo\") pod \"a36c9b31-8ab0-449f-8fb4-87a702ab7abf\" (UID: \"a36c9b31-8ab0-449f-8fb4-87a702ab7abf\") "
Nov 03 18:15:01.704725 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.704701    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede64d2a-75d4-4d21-8592-1680fb138e4e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede64d2a-75d4-4d21-8592-1680fb138e4e-workload-socket\") pod \"ede64d2a-75d4-4d21-8592-1680fb138e4e\" (UID: \"ede64d2a-75d4-4d21-8592-1680fb138e4e\") "
Nov 03 18:15:01.704852 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.704741    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-workload-certs\") pod \"c4428ff6-1929-4fb0-82ee-ab6a785ca9e5\" (UID: \"c4428ff6-1929-4fb0-82ee-ab6a785ca9e5\") "
Nov 03 18:15:01.704852 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.704774    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-kube-api-access-56w4t\" (UniqueName: \"kubernetes.io/projected/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-kube-api-access-56w4t\") pod \"c4428ff6-1929-4fb0-82ee-ab6a785ca9e5\" (UID: \"c4428ff6-1929-4fb0-82ee-ab6a785ca9e5\") "
Nov 03 18:15:01.704852 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.704808    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ede64d2a-75d4-4d21-8592-1680fb138e4e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ede64d2a-75d4-4d21-8592-1680fb138e4e-istiod-ca-cert\") pod \"ede64d2a-75d4-4d21-8592-1680fb138e4e\" (UID: \"ede64d2a-75d4-4d21-8592-1680fb138e4e\") "
Nov 03 18:15:01.704852 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.704843    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-istiod-ca-cert\") pod \"a36c9b31-8ab0-449f-8fb4-87a702ab7abf\" (UID: \"a36c9b31-8ab0-449f-8fb4-87a702ab7abf\") "
Nov 03 18:15:01.705091 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.704871    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede64d2a-75d4-4d21-8592-1680fb138e4e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ede64d2a-75d4-4d21-8592-1680fb138e4e-workload-certs\") pod \"ede64d2a-75d4-4d21-8592-1680fb138e4e\" (UID: \"ede64d2a-75d4-4d21-8592-1680fb138e4e\") "
Nov 03 18:15:01.705091 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.704937    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-credential-socket\") pod \"c4428ff6-1929-4fb0-82ee-ab6a785ca9e5\" (UID: \"c4428ff6-1929-4fb0-82ee-ab6a785ca9e5\") "
Nov 03 18:15:01.705091 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.704973    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-istio-envoy\") pod \"a36c9b31-8ab0-449f-8fb4-87a702ab7abf\" (UID: \"a36c9b31-8ab0-449f-8fb4-87a702ab7abf\") "
Nov 03 18:15:01.705091 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.705019    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-istio-token\" (UniqueName: \"kubernetes.io/projected/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-istio-token\") pod \"c4428ff6-1929-4fb0-82ee-ab6a785ca9e5\" (UID: \"c4428ff6-1929-4fb0-82ee-ab6a785ca9e5\") "
Nov 03 18:15:01.705091 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.705087    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-istio-data\") pod \"c4428ff6-1929-4fb0-82ee-ab6a785ca9e5\" (UID: \"c4428ff6-1929-4fb0-82ee-ab6a785ca9e5\") "
Nov 03 18:15:01.708007 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.705131    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-istio-envoy\") pod \"c4428ff6-1929-4fb0-82ee-ab6a785ca9e5\" (UID: \"c4428ff6-1929-4fb0-82ee-ab6a785ca9e5\") "
Nov 03 18:15:01.708007 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.705178    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-workload-socket\") pod \"c4428ff6-1929-4fb0-82ee-ab6a785ca9e5\" (UID: \"c4428ff6-1929-4fb0-82ee-ab6a785ca9e5\") "
Nov 03 18:15:01.708007 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.705236    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-credential-socket\") pod \"a36c9b31-8ab0-449f-8fb4-87a702ab7abf\" (UID: \"a36c9b31-8ab0-449f-8fb4-87a702ab7abf\") "
Nov 03 18:15:01.708007 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.705292    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-istio-podinfo\") pod \"c4428ff6-1929-4fb0-82ee-ab6a785ca9e5\" (UID: \"c4428ff6-1929-4fb0-82ee-ab6a785ca9e5\") "
Nov 03 18:15:01.708007 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.705334    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede64d2a-75d4-4d21-8592-1680fb138e4e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ede64d2a-75d4-4d21-8592-1680fb138e4e-istio-data\") pod \"ede64d2a-75d4-4d21-8592-1680fb138e4e\" (UID: \"ede64d2a-75d4-4d21-8592-1680fb138e4e\") "
Nov 03 18:15:01.708007 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.705361    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ede64d2a-75d4-4d21-8592-1680fb138e4e-istio-token\" (UniqueName: \"kubernetes.io/projected/ede64d2a-75d4-4d21-8592-1680fb138e4e-istio-token\") pod \"ede64d2a-75d4-4d21-8592-1680fb138e4e\" (UID: \"ede64d2a-75d4-4d21-8592-1680fb138e4e\") "
Nov 03 18:15:01.708007 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.705406    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede64d2a-75d4-4d21-8592-1680fb138e4e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede64d2a-75d4-4d21-8592-1680fb138e4e-credential-socket\") pod \"ede64d2a-75d4-4d21-8592-1680fb138e4e\" (UID: \"ede64d2a-75d4-4d21-8592-1680fb138e4e\") "
Nov 03 18:15:01.708007 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.705452    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-istio-data\") pod \"a36c9b31-8ab0-449f-8fb4-87a702ab7abf\" (UID: \"a36c9b31-8ab0-449f-8fb4-87a702ab7abf\") "
Nov 03 18:15:01.708007 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.705478    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-istio-token\" (UniqueName: \"kubernetes.io/projected/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-istio-token\") pod \"a36c9b31-8ab0-449f-8fb4-87a702ab7abf\" (UID: \"a36c9b31-8ab0-449f-8fb4-87a702ab7abf\") "
Nov 03 18:15:01.708007 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.705517    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-workload-socket\") pod \"a36c9b31-8ab0-449f-8fb4-87a702ab7abf\" (UID: \"a36c9b31-8ab0-449f-8fb4-87a702ab7abf\") "
Nov 03 18:15:01.708007 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.705559    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-kube-api-access-2ptf5\" (UniqueName: \"kubernetes.io/projected/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-kube-api-access-2ptf5\") pod \"a36c9b31-8ab0-449f-8fb4-87a702ab7abf\" (UID: \"a36c9b31-8ab0-449f-8fb4-87a702ab7abf\") "
Nov 03 18:15:01.708007 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.705689    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-workload-certs\") pod \"a36c9b31-8ab0-449f-8fb4-87a702ab7abf\" (UID: \"a36c9b31-8ab0-449f-8fb4-87a702ab7abf\") "
Nov 03 18:15:01.708007 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.705727    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ede64d2a-75d4-4d21-8592-1680fb138e4e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ede64d2a-75d4-4d21-8592-1680fb138e4e-istio-podinfo\") pod \"ede64d2a-75d4-4d21-8592-1680fb138e4e\" (UID: \"ede64d2a-75d4-4d21-8592-1680fb138e4e\") "
Nov 03 18:15:01.708007 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.705795    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ede64d2a-75d4-4d21-8592-1680fb138e4e-kube-api-access-2s44q\" (UniqueName: \"kubernetes.io/projected/ede64d2a-75d4-4d21-8592-1680fb138e4e-kube-api-access-2s44q\") pod \"ede64d2a-75d4-4d21-8592-1680fb138e4e\" (UID: \"ede64d2a-75d4-4d21-8592-1680fb138e4e\") "
Nov 03 18:15:01.708007 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.705901    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-istiod-ca-cert\") pod \"c4428ff6-1929-4fb0-82ee-ab6a785ca9e5\" (UID: \"c4428ff6-1929-4fb0-82ee-ab6a785ca9e5\") "
Nov 03 18:15:01.708007 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.707155    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-istiod-ca-cert" pod "c4428ff6-1929-4fb0-82ee-ab6a785ca9e5" (UID: "c4428ff6-1929-4fb0-82ee-ab6a785ca9e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:01.708007 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.707403    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede64d2a-75d4-4d21-8592-1680fb138e4e-workload-socket" pod "ede64d2a-75d4-4d21-8592-1680fb138e4e" (UID: "ede64d2a-75d4-4d21-8592-1680fb138e4e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:01.708007 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.707561    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-workload-certs" pod "c4428ff6-1929-4fb0-82ee-ab6a785ca9e5" (UID: "c4428ff6-1929-4fb0-82ee-ab6a785ca9e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:01.708940 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.708146    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede64d2a-75d4-4d21-8592-1680fb138e4e-workload-certs" pod "ede64d2a-75d4-4d21-8592-1680fb138e4e" (UID: "ede64d2a-75d4-4d21-8592-1680fb138e4e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:01.708940 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.708454    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-istiod-ca-cert" pod "a36c9b31-8ab0-449f-8fb4-87a702ab7abf" (UID: "a36c9b31-8ab0-449f-8fb4-87a702ab7abf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:01.708940 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.708761    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-workload-certs" pod "a36c9b31-8ab0-449f-8fb4-87a702ab7abf" (UID: "a36c9b31-8ab0-449f-8fb4-87a702ab7abf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:01.709097 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.709040    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ede64d2a-75d4-4d21-8592-1680fb138e4e-istiod-ca-cert" pod "ede64d2a-75d4-4d21-8592-1680fb138e4e" (UID: "ede64d2a-75d4-4d21-8592-1680fb138e4e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:01.709364 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.709339    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-workload-socket" pod "a36c9b31-8ab0-449f-8fb4-87a702ab7abf" (UID: "a36c9b31-8ab0-449f-8fb4-87a702ab7abf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:01.721350 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.721310    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede64d2a-75d4-4d21-8592-1680fb138e4e-credential-socket" pod "ede64d2a-75d4-4d21-8592-1680fb138e4e" (UID: "ede64d2a-75d4-4d21-8592-1680fb138e4e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:01.721549 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.721510    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-workload-socket" pod "c4428ff6-1929-4fb0-82ee-ab6a785ca9e5" (UID: "c4428ff6-1929-4fb0-82ee-ab6a785ca9e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:01.721974 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.721947    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-istio-data" pod "c4428ff6-1929-4fb0-82ee-ab6a785ca9e5" (UID: "c4428ff6-1929-4fb0-82ee-ab6a785ca9e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:01.722291 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.722207    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-credential-socket" pod "c4428ff6-1929-4fb0-82ee-ab6a785ca9e5" (UID: "c4428ff6-1929-4fb0-82ee-ab6a785ca9e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:01.722291 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.722233    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede64d2a-75d4-4d21-8592-1680fb138e4e-istio-data" pod "ede64d2a-75d4-4d21-8592-1680fb138e4e" (UID: "ede64d2a-75d4-4d21-8592-1680fb138e4e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:01.722291 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.722262    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-credential-socket" pod "a36c9b31-8ab0-449f-8fb4-87a702ab7abf" (UID: "a36c9b31-8ab0-449f-8fb4-87a702ab7abf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:01.722501 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.722321    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-istio-data" pod "a36c9b31-8ab0-449f-8fb4-87a702ab7abf" (UID: "a36c9b31-8ab0-449f-8fb4-87a702ab7abf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:01.730854 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.730788    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede64d2a-75d4-4d21-8592-1680fb138e4e-istio-envoy" pod "ede64d2a-75d4-4d21-8592-1680fb138e4e" (UID: "ede64d2a-75d4-4d21-8592-1680fb138e4e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:01.731038 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.730870    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ede64d2a-75d4-4d21-8592-1680fb138e4e-kube-api-access-2s44q" pod "ede64d2a-75d4-4d21-8592-1680fb138e4e" (UID: "ede64d2a-75d4-4d21-8592-1680fb138e4e"). InnerVolumeSpecName "kube-api-access-2s44q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:01.731344 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.731307    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-istio-podinfo" pod "a36c9b31-8ab0-449f-8fb4-87a702ab7abf" (UID: "a36c9b31-8ab0-449f-8fb4-87a702ab7abf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:15:01.734265 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.734218    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-istio-token" pod "c4428ff6-1929-4fb0-82ee-ab6a785ca9e5" (UID: "c4428ff6-1929-4fb0-82ee-ab6a785ca9e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:01.735937 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.735889    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-kube-api-access-2ptf5" pod "a36c9b31-8ab0-449f-8fb4-87a702ab7abf" (UID: "a36c9b31-8ab0-449f-8fb4-87a702ab7abf"). InnerVolumeSpecName "kube-api-access-2ptf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:01.737954 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.736513    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ede64d2a-75d4-4d21-8592-1680fb138e4e-istio-podinfo" pod "ede64d2a-75d4-4d21-8592-1680fb138e4e" (UID: "ede64d2a-75d4-4d21-8592-1680fb138e4e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:15:01.737954 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.736572    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-kube-api-access-56w4t" pod "c4428ff6-1929-4fb0-82ee-ab6a785ca9e5" (UID: "c4428ff6-1929-4fb0-82ee-ab6a785ca9e5"). InnerVolumeSpecName "kube-api-access-56w4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:01.737954 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.736587    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-istio-token" pod "a36c9b31-8ab0-449f-8fb4-87a702ab7abf" (UID: "a36c9b31-8ab0-449f-8fb4-87a702ab7abf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:01.809213 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.809169    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ede64d2a-75d4-4d21-8592-1680fb138e4e-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:01.809213 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.809214    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:01.809439 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.809231    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede64d2a-75d4-4d21-8592-1680fb138e4e-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:01.809439 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.809246    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:01.809439 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.809261    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56w4t\" (UniqueName: \"kubernetes.io/projected/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-kube-api-access-56w4t\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:01.809439 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.809278    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ede64d2a-75d4-4d21-8592-1680fb138e4e-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:01.809439 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.809292    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:01.809439 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.809306    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ede64d2a-75d4-4d21-8592-1680fb138e4e-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:01.809439 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.809320    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:01.809439 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.809333    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:01.809439 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.809347    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:01.809439 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.809362    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:01.809439 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.809376    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:01.809439 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.809389    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ede64d2a-75d4-4d21-8592-1680fb138e4e-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:01.809439 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.809402    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede64d2a-75d4-4d21-8592-1680fb138e4e-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:01.809439 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.809416    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:01.809439 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.809429    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:01.809439 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.809441    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:01.810082 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.809457    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ptf5\" (UniqueName: \"kubernetes.io/projected/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-kube-api-access-2ptf5\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:01.810082 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.809471    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:01.810082 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.809484    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ede64d2a-75d4-4d21-8592-1680fb138e4e-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:01.810082 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.809498    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s44q\" (UniqueName: \"kubernetes.io/projected/ede64d2a-75d4-4d21-8592-1680fb138e4e-kube-api-access-2s44q\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:01.810082 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.809531    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:01.824284 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.824076    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-istio-podinfo" pod "c4428ff6-1929-4fb0-82ee-ab6a785ca9e5" (UID: "c4428ff6-1929-4fb0-82ee-ab6a785ca9e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:15:01.824455 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.824395    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-istio-envoy" pod "a36c9b31-8ab0-449f-8fb4-87a702ab7abf" (UID: "a36c9b31-8ab0-449f-8fb4-87a702ab7abf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:01.824538 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.824520    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ede64d2a-75d4-4d21-8592-1680fb138e4e-istio-token" pod "ede64d2a-75d4-4d21-8592-1680fb138e4e" (UID: "ede64d2a-75d4-4d21-8592-1680fb138e4e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:01.824800 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.824757    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-istio-envoy" pod "c4428ff6-1929-4fb0-82ee-ab6a785ca9e5" (UID: "c4428ff6-1929-4fb0-82ee-ab6a785ca9e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:01.891323 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.891258    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c1b2d2-29895495-k5x2m" podStartSLOduration=1.891240732 podStartE2EDuration="1.891240732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:15:01.89121609 +0000 UTC m=+1740.907414159" watchObservedRunningTime="2026-11-03 18:15:01.891240732 +0000 UTC m=+1740.907438801"
Nov 03 18:15:01.911052 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.911009    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a36c9b31-8ab0-449f-8fb4-87a702ab7abf-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:01.911052 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.911045    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:01.911052 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.911056    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:01.911052 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:01.911066    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ede64d2a-75d4-4d21-8592-1680fb138e4e-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:02.037704 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:02.035964    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a86228-7nnxk"]
Nov 03 18:15:02.037704 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:02.037026    5315 cpu_manager.go:414] "containerMap: removing container" podUID="c4428ff6-1929-4fb0-82ee-ab6a785ca9e5" containerName="istio-proxy"
Nov 03 18:15:02.037704 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:02.037048    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4428ff6-1929-4fb0-82ee-ab6a785ca9e5" containerName="istio-proxy"
Nov 03 18:15:02.037704 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:02.037066    5315 cpu_manager.go:414] "containerMap: removing container" podUID="a36c9b31-8ab0-449f-8fb4-87a702ab7abf" containerName="istio-proxy"
Nov 03 18:15:02.037704 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:02.037073    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a36c9b31-8ab0-449f-8fb4-87a702ab7abf" containerName="istio-proxy"
Nov 03 18:15:02.037704 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:02.037085    5315 cpu_manager.go:414] "containerMap: removing container" podUID="ede64d2a-75d4-4d21-8592-1680fb138e4e" containerName="istio-proxy"
Nov 03 18:15:02.037704 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:02.037094    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ede64d2a-75d4-4d21-8592-1680fb138e4e" containerName="istio-proxy"
Nov 03 18:15:02.037704 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:02.037284    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4428ff6-1929-4fb0-82ee-ab6a785ca9e5" containerName="istio-proxy"
Nov 03 18:15:02.037704 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:02.037295    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="a36c9b31-8ab0-449f-8fb4-87a702ab7abf" containerName="istio-proxy"
Nov 03 18:15:02.037704 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:02.037304    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="ede64d2a-75d4-4d21-8592-1680fb138e4e" containerName="istio-proxy"
Nov 03 18:15:02.041158 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:02.041114    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a86228-7nnxk"
Nov 03 18:15:02.091368 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:02.091332    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:15:02.094851 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:02.094820    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a86228-7nnxk"]
Nov 03 18:15:02.158452 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:02.158414    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb"]
Nov 03 18:15:02.162630 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:02.162585    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8mstb"]
Nov 03 18:15:02.215256 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:02.215157    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-a86228-7nnxk\" (UID: \"658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a86228-7nnxk"
Nov 03 18:15:02.215415 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:02.215355    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-a86228-7nnxk\" (UID: \"658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a86228-7nnxk"
Nov 03 18:15:02.215716 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:02.215694    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66mfm\" (UniqueName: \"kubernetes.io/projected/658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51-kube-api-access-66mfm\") pod \"maas-api-revoke-keys-e2e-worker-w4-a86228-7nnxk\" (UID: \"658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a86228-7nnxk"
Nov 03 18:15:02.317198 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:02.317158    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-a86228-7nnxk\" (UID: \"658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a86228-7nnxk"
Nov 03 18:15:02.317643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:02.317279    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66mfm\" (UniqueName: \"kubernetes.io/projected/658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51-kube-api-access-66mfm\") pod \"maas-api-revoke-keys-e2e-worker-w4-a86228-7nnxk\" (UID: \"658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a86228-7nnxk"
Nov 03 18:15:02.317643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:02.317405    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-a86228-7nnxk\" (UID: \"658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a86228-7nnxk"
Nov 03 18:15:02.317843 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:02.317821    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-a86228-7nnxk\" (UID: \"658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a86228-7nnxk"
Nov 03 18:15:02.319780 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:02.319758    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-a86228-7nnxk\" (UID: \"658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a86228-7nnxk"
Nov 03 18:15:02.374886 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:02.374855    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66mfm\" (UniqueName: \"kubernetes.io/projected/658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51-kube-api-access-66mfm\") pod \"maas-api-revoke-keys-e2e-worker-w4-a86228-7nnxk\" (UID: \"658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a86228-7nnxk"
Nov 03 18:15:02.407538 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:02.407503    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl"]
Nov 03 18:15:02.411768 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:02.411735    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8ltsl"]
Nov 03 18:15:02.513558 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:02.513456    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv"]
Nov 03 18:15:02.519575 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:02.519540    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-zn5lv"]
Nov 03 18:15:02.661014 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:02.660985    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a86228-7nnxk"
Nov 03 18:15:02.671349 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:02.671317    5315 generic.go:356] "Generic (PLEG): container finished" podID="26d8c8d7-89c0-47af-b90c-8b0677f71886" containerID="1e291dc9f5e11243ffb5e58992feea7cdfc0f94c2e8c791788ed66a0562990cc" exitCode=60
Nov 03 18:15:02.671468 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:02.671399    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-80aca1-29895495-qvtbt" event={"ID":"26d8c8d7-89c0-47af-b90c-8b0677f71886","Type":"ContainerDied","Data":"1e291dc9f5e11243ffb5e58992feea7cdfc0f94c2e8c791788ed66a0562990cc"}
Nov 03 18:15:02.671866 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:02.671836    5315 scope.go:122] "RemoveContainer" containerID="1e291dc9f5e11243ffb5e58992feea7cdfc0f94c2e8c791788ed66a0562990cc"
Nov 03 18:15:02.673318 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:02.673288    5315 generic.go:356] "Generic (PLEG): container finished" podID="e31a2c86-cff2-49ed-8f40-4020dfef4808" containerID="e8c8447cdb951aec41b0824da72e63ec872bd1682e08f1ef14683e55e6b11d64" exitCode=60
Nov 03 18:15:02.673438 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:02.673374    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-jgzv4" event={"ID":"e31a2c86-cff2-49ed-8f40-4020dfef4808","Type":"ContainerDied","Data":"e8c8447cdb951aec41b0824da72e63ec872bd1682e08f1ef14683e55e6b11d64"}
Nov 03 18:15:02.673734 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:02.673672    5315 scope.go:122] "RemoveContainer" containerID="e8c8447cdb951aec41b0824da72e63ec872bd1682e08f1ef14683e55e6b11d64"
Nov 03 18:15:02.675855 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:02.675279    5315 generic.go:356] "Generic (PLEG): container finished" podID="5d04a4ae-44e9-4f14-ada5-855c1efe7539" containerID="815dcfaf22e6096d7e5a6d35c7fb7cb4ca6531903719739a01672f2bdf712bbe" exitCode=60
Nov 03 18:15:02.675855 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:02.675385    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895495-r68gb" event={"ID":"5d04a4ae-44e9-4f14-ada5-855c1efe7539","Type":"ContainerDied","Data":"815dcfaf22e6096d7e5a6d35c7fb7cb4ca6531903719739a01672f2bdf712bbe"}
Nov 03 18:15:02.675855 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:02.675656    5315 scope.go:122] "RemoveContainer" containerID="815dcfaf22e6096d7e5a6d35c7fb7cb4ca6531903719739a01672f2bdf712bbe"
Nov 03 18:15:02.678194 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:02.678177    5315 generic.go:356] "Generic (PLEG): container finished" podID="8d8504d9-e3e9-4542-8e76-3dbd404ed89f" containerID="d6b4e542c94c5546e7ba207998ac8c0f7c59af325b3b5c9a055169d0522c6542" exitCode=60
Nov 03 18:15:02.678282 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:02.678232    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7a7c76-29895495-lhj7q" event={"ID":"8d8504d9-e3e9-4542-8e76-3dbd404ed89f","Type":"ContainerDied","Data":"d6b4e542c94c5546e7ba207998ac8c0f7c59af325b3b5c9a055169d0522c6542"}
Nov 03 18:15:02.678470 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:02.678455    5315 scope.go:122] "RemoveContainer" containerID="d6b4e542c94c5546e7ba207998ac8c0f7c59af325b3b5c9a055169d0522c6542"
Nov 03 18:15:02.679736 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:02.679718    5315 generic.go:356] "Generic (PLEG): container finished" podID="54bd478e-0685-4ec9-8aa0-5c3bd73c56c2" containerID="1a38ecd489dc8610a155205fdf844a0bdc020c0c15cd5681fee7241f8df532ea" exitCode=60
Nov 03 18:15:02.679822 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:02.679799    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895495-shccb" event={"ID":"54bd478e-0685-4ec9-8aa0-5c3bd73c56c2","Type":"ContainerDied","Data":"1a38ecd489dc8610a155205fdf844a0bdc020c0c15cd5681fee7241f8df532ea"}
Nov 03 18:15:02.679979 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:02.679960    5315 scope.go:122] "RemoveContainer" containerID="1a38ecd489dc8610a155205fdf844a0bdc020c0c15cd5681fee7241f8df532ea"
Nov 03 18:15:02.681537 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:02.681518    5315 generic.go:356] "Generic (PLEG): container finished" podID="e3a2676b-0fd6-4c39-9ed7-d468706aa9e9" containerID="8a3016b2945d5b575495d0ddcb52cbd7ce892b95f35ff2f011684eb1c229e65e" exitCode=60
Nov 03 18:15:02.681646 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:02.681553    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c1b2d2-29895495-k5x2m" event={"ID":"e3a2676b-0fd6-4c39-9ed7-d468706aa9e9","Type":"ContainerDied","Data":"8a3016b2945d5b575495d0ddcb52cbd7ce892b95f35ff2f011684eb1c229e65e"}
Nov 03 18:15:02.681777 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:02.681762    5315 scope.go:122] "RemoveContainer" containerID="8a3016b2945d5b575495d0ddcb52cbd7ce892b95f35ff2f011684eb1c229e65e"
Nov 03 18:15:03.073443 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:03.073407    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a86228-7nnxk"]
Nov 03 18:15:03.105484 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:03.105442    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a36c9b31-8ab0-449f-8fb4-87a702ab7abf" path="/var/lib/kubelet/pods/a36c9b31-8ab0-449f-8fb4-87a702ab7abf/volumes"
Nov 03 18:15:03.106324 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:03.106301    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4428ff6-1929-4fb0-82ee-ab6a785ca9e5" path="/var/lib/kubelet/pods/c4428ff6-1929-4fb0-82ee-ab6a785ca9e5/volumes"
Nov 03 18:15:03.107011 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:03.106992    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ede64d2a-75d4-4d21-8592-1680fb138e4e" path="/var/lib/kubelet/pods/ede64d2a-75d4-4d21-8592-1680fb138e4e/volumes"
Nov 03 18:15:03.691876 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:03.691765    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c1b2d2-29895495-k5x2m" event={"ID":"e3a2676b-0fd6-4c39-9ed7-d468706aa9e9","Type":"ContainerStarted","Data":"a6b7bcc438fc29b822204297cb9755cdaaf0cad240daf00031cacb69c32bde80"}
Nov 03 18:15:03.693808 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:03.693776    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-80aca1-29895495-qvtbt" event={"ID":"26d8c8d7-89c0-47af-b90c-8b0677f71886","Type":"ContainerStarted","Data":"d8fa4764e246686dcba4005f14924b14ff01cc2bae3d23063475ee329e06b8db"}
Nov 03 18:15:03.695134 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:03.695108    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a86228-7nnxk" event={"ID":"658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51","Type":"ContainerStarted","Data":"16e9dfe3f571d3f8c70b06a04b6d4bf4ab5eaf28b7b5fb4b8bcbd80ec3cd0fa0"}
Nov 03 18:15:03.696876 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:03.696853    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-jgzv4" event={"ID":"e31a2c86-cff2-49ed-8f40-4020dfef4808","Type":"ContainerStarted","Data":"641a9509da3ff55fadbd9e96744036800cd32b606e9ad3de2ee1782a9abb0e8f"}
Nov 03 18:15:03.698956 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:03.698927    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895495-r68gb" event={"ID":"5d04a4ae-44e9-4f14-ada5-855c1efe7539","Type":"ContainerStarted","Data":"6732a4b4ec6cd747d6a75f3808ff1431e120d5c9f95c0cebf37f4b0903d8e1c9"}
Nov 03 18:15:03.700948 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:03.700926    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7a7c76-29895495-lhj7q" event={"ID":"8d8504d9-e3e9-4542-8e76-3dbd404ed89f","Type":"ContainerStarted","Data":"821b86c994e38da0552da259b84afa48193aa291cc841f32c7e6bdd29b73d9b6"}
Nov 03 18:15:03.702849 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:03.702827    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895495-shccb" event={"ID":"54bd478e-0685-4ec9-8aa0-5c3bd73c56c2","Type":"ContainerStarted","Data":"3ce363f886ede99f0c20907b1c1c353410392b5fa53d5f07edc13c66d405a918"}
Nov 03 18:15:03.913547 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:03.913492    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-jgzv4" podStartSLOduration=3.913477073 podStartE2EDuration="3.913477073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:15:03.911205578 +0000 UTC m=+1742.927403646" watchObservedRunningTime="2026-11-03 18:15:03.913477073 +0000 UTC m=+1742.929675136"
Nov 03 18:15:04.020144 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:04.020084    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-80aca1-29895495-qvtbt" podStartSLOduration=4.020070351 podStartE2EDuration="4.020070351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:15:04.01791415 +0000 UTC m=+1743.034112219" watchObservedRunningTime="2026-11-03 18:15:04.020070351 +0000 UTC m=+1743.036268421"
Nov 03 18:15:04.127338 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:04.127284    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895495-r68gb" podStartSLOduration=4.127269228 podStartE2EDuration="4.127269228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:15:04.12588557 +0000 UTC m=+1743.142083638" watchObservedRunningTime="2026-11-03 18:15:04.127269228 +0000 UTC m=+1743.143467296"
Nov 03 18:15:04.234483 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:04.234424    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895495-shccb" podStartSLOduration=4.234407511 podStartE2EDuration="4.234407511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:15:04.233085681 +0000 UTC m=+1743.249283750" watchObservedRunningTime="2026-11-03 18:15:04.234407511 +0000 UTC m=+1743.250605580"
Nov 03 18:15:04.342963 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:04.342899    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7a7c76-29895495-lhj7q" podStartSLOduration=4.342884928 podStartE2EDuration="4.342884928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:15:04.340875694 +0000 UTC m=+1743.357073764" watchObservedRunningTime="2026-11-03 18:15:04.342884928 +0000 UTC m=+1743.359082997"
Nov 03 18:15:04.536405 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:04.536366    5315 scope.go:122] "RemoveContainer" containerID="8bb6a8719a14e5b36a0e8736898d047f4fcb42e2a4258f42617d7020447b895f"
Nov 03 18:15:04.547653 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:04.547601    5315 scope.go:122] "RemoveContainer" containerID="845ad1e41d79011c112e61c1577535866fb7a3612fa3375e2a4875e9d30e65fb"
Nov 03 18:15:04.560937 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:04.560910    5315 scope.go:122] "RemoveContainer" containerID="581dfc7b11bfd350ef5d7bf45ae44da894484af675859a287adaa3cf2ca10a8f"
Nov 03 18:15:04.571529 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:04.571503    5315 scope.go:122] "RemoveContainer" containerID="0313a23b322ac08662841f6212b191959e8905ecde57612eca1e2535334072f0"
Nov 03 18:15:04.714397 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:04.714367    5315 generic.go:356] "Generic (PLEG): container finished" podID="e31a2c86-cff2-49ed-8f40-4020dfef4808" containerID="641a9509da3ff55fadbd9e96744036800cd32b606e9ad3de2ee1782a9abb0e8f" exitCode=60
Nov 03 18:15:04.714880 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:04.714453    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-jgzv4" event={"ID":"e31a2c86-cff2-49ed-8f40-4020dfef4808","Type":"ContainerDied","Data":"641a9509da3ff55fadbd9e96744036800cd32b606e9ad3de2ee1782a9abb0e8f"}
Nov 03 18:15:04.714880 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:04.714558    5315 scope.go:122] "RemoveContainer" containerID="e8c8447cdb951aec41b0824da72e63ec872bd1682e08f1ef14683e55e6b11d64"
Nov 03 18:15:04.714880 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:04.714864    5315 scope.go:122] "RemoveContainer" containerID="641a9509da3ff55fadbd9e96744036800cd32b606e9ad3de2ee1782a9abb0e8f"
Nov 03 18:15:04.715338 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:15:04.715313    5315 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895495-jgzv4_odh-ai-gateway-infra(e31a2c86-cff2-49ed-8f40-4020dfef4808)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-jgzv4" podUID="e31a2c86-cff2-49ed-8f40-4020dfef4808"
Nov 03 18:15:04.716819 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:04.716797    5315 generic.go:356] "Generic (PLEG): container finished" podID="5d04a4ae-44e9-4f14-ada5-855c1efe7539" containerID="6732a4b4ec6cd747d6a75f3808ff1431e120d5c9f95c0cebf37f4b0903d8e1c9" exitCode=60
Nov 03 18:15:04.716921 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:04.716879    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895495-r68gb" event={"ID":"5d04a4ae-44e9-4f14-ada5-855c1efe7539","Type":"ContainerDied","Data":"6732a4b4ec6cd747d6a75f3808ff1431e120d5c9f95c0cebf37f4b0903d8e1c9"}
Nov 03 18:15:04.717216 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:04.717200    5315 scope.go:122] "RemoveContainer" containerID="6732a4b4ec6cd747d6a75f3808ff1431e120d5c9f95c0cebf37f4b0903d8e1c9"
Nov 03 18:15:04.717586 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:15:04.717551    5315 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-a86228-29895495-r68gb_odh-ai-gateway-infra(5d04a4ae-44e9-4f14-ada5-855c1efe7539)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895495-r68gb" podUID="5d04a4ae-44e9-4f14-ada5-855c1efe7539"
Nov 03 18:15:04.719361 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:04.719333    5315 generic.go:356] "Generic (PLEG): container finished" podID="8d8504d9-e3e9-4542-8e76-3dbd404ed89f" containerID="821b86c994e38da0552da259b84afa48193aa291cc841f32c7e6bdd29b73d9b6" exitCode=60
Nov 03 18:15:04.719530 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:04.719414    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7a7c76-29895495-lhj7q" event={"ID":"8d8504d9-e3e9-4542-8e76-3dbd404ed89f","Type":"ContainerDied","Data":"821b86c994e38da0552da259b84afa48193aa291cc841f32c7e6bdd29b73d9b6"}
Nov 03 18:15:04.719783 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:04.719764    5315 scope.go:122] "RemoveContainer" containerID="821b86c994e38da0552da259b84afa48193aa291cc841f32c7e6bdd29b73d9b6"
Nov 03 18:15:04.720179 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:15:04.720158    5315 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-7a7c76-29895495-lhj7q_odh-ai-gateway-infra(8d8504d9-e3e9-4542-8e76-3dbd404ed89f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7a7c76-29895495-lhj7q" podUID="8d8504d9-e3e9-4542-8e76-3dbd404ed89f"
Nov 03 18:15:04.721483 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:04.721461    5315 generic.go:356] "Generic (PLEG): container finished" podID="54bd478e-0685-4ec9-8aa0-5c3bd73c56c2" containerID="3ce363f886ede99f0c20907b1c1c353410392b5fa53d5f07edc13c66d405a918" exitCode=60
Nov 03 18:15:04.721559 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:04.721542    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895495-shccb" event={"ID":"54bd478e-0685-4ec9-8aa0-5c3bd73c56c2","Type":"ContainerDied","Data":"3ce363f886ede99f0c20907b1c1c353410392b5fa53d5f07edc13c66d405a918"}
Nov 03 18:15:04.721867 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:04.721849    5315 scope.go:122] "RemoveContainer" containerID="3ce363f886ede99f0c20907b1c1c353410392b5fa53d5f07edc13c66d405a918"
Nov 03 18:15:04.722227 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:15:04.722205    5315 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-f79c02-29895495-shccb_odh-ai-gateway-infra(54bd478e-0685-4ec9-8aa0-5c3bd73c56c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895495-shccb" podUID="54bd478e-0685-4ec9-8aa0-5c3bd73c56c2"
Nov 03 18:15:04.723846 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:04.723823    5315 generic.go:356] "Generic (PLEG): container finished" podID="e3a2676b-0fd6-4c39-9ed7-d468706aa9e9" containerID="a6b7bcc438fc29b822204297cb9755cdaaf0cad240daf00031cacb69c32bde80" exitCode=60
Nov 03 18:15:04.723940 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:04.723883    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c1b2d2-29895495-k5x2m" event={"ID":"e3a2676b-0fd6-4c39-9ed7-d468706aa9e9","Type":"ContainerDied","Data":"a6b7bcc438fc29b822204297cb9755cdaaf0cad240daf00031cacb69c32bde80"}
Nov 03 18:15:04.724196 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:04.724181    5315 scope.go:122] "RemoveContainer" containerID="a6b7bcc438fc29b822204297cb9755cdaaf0cad240daf00031cacb69c32bde80"
Nov 03 18:15:04.724518 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:15:04.724493    5315 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-c1b2d2-29895495-k5x2m_odh-ai-gateway-infra(e3a2676b-0fd6-4c39-9ed7-d468706aa9e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c1b2d2-29895495-k5x2m" podUID="e3a2676b-0fd6-4c39-9ed7-d468706aa9e9"
Nov 03 18:15:04.726224 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:04.726202    5315 generic.go:356] "Generic (PLEG): container finished" podID="26d8c8d7-89c0-47af-b90c-8b0677f71886" containerID="d8fa4764e246686dcba4005f14924b14ff01cc2bae3d23063475ee329e06b8db" exitCode=60
Nov 03 18:15:04.726331 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:04.726306    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-80aca1-29895495-qvtbt" event={"ID":"26d8c8d7-89c0-47af-b90c-8b0677f71886","Type":"ContainerDied","Data":"d8fa4764e246686dcba4005f14924b14ff01cc2bae3d23063475ee329e06b8db"}
Nov 03 18:15:04.726573 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:04.726550    5315 scope.go:122] "RemoveContainer" containerID="d8fa4764e246686dcba4005f14924b14ff01cc2bae3d23063475ee329e06b8db"
Nov 03 18:15:04.726913 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:15:04.726888    5315 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-80aca1-29895495-qvtbt_odh-ai-gateway-infra(26d8c8d7-89c0-47af-b90c-8b0677f71886)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-80aca1-29895495-qvtbt" podUID="26d8c8d7-89c0-47af-b90c-8b0677f71886"
Nov 03 18:15:04.728809 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:04.728785    5315 generic.go:356] "Generic (PLEG): container finished" podID="658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51" containerID="72dc22a814a81e21808ff59f3a138415350510459cd0c35b9e2058bdec025047" exitCode=0
Nov 03 18:15:04.728917 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:04.728811    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a86228-7nnxk" event={"ID":"658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51","Type":"ContainerDied","Data":"72dc22a814a81e21808ff59f3a138415350510459cd0c35b9e2058bdec025047"}
Nov 03 18:15:04.734331 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:04.734315    5315 scope.go:122] "RemoveContainer" containerID="815dcfaf22e6096d7e5a6d35c7fb7cb4ca6531903719739a01672f2bdf712bbe"
Nov 03 18:15:04.748600 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:04.748477    5315 scope.go:122] "RemoveContainer" containerID="d6b4e542c94c5546e7ba207998ac8c0f7c59af325b3b5c9a055169d0522c6542"
Nov 03 18:15:04.764559 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:04.764534    5315 scope.go:122] "RemoveContainer" containerID="1a38ecd489dc8610a155205fdf844a0bdc020c0c15cd5681fee7241f8df532ea"
Nov 03 18:15:04.778520 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:04.778495    5315 scope.go:122] "RemoveContainer" containerID="8a3016b2945d5b575495d0ddcb52cbd7ce892b95f35ff2f011684eb1c229e65e"
Nov 03 18:15:04.792035 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:04.792007    5315 scope.go:122] "RemoveContainer" containerID="1e291dc9f5e11243ffb5e58992feea7cdfc0f94c2e8c791788ed66a0562990cc"
Nov 03 18:15:05.738871 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:05.738841    5315 scope.go:122] "RemoveContainer" containerID="a6b7bcc438fc29b822204297cb9755cdaaf0cad240daf00031cacb69c32bde80"
Nov 03 18:15:05.739299 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:15:05.739182    5315 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-c1b2d2-29895495-k5x2m_odh-ai-gateway-infra(e3a2676b-0fd6-4c39-9ed7-d468706aa9e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c1b2d2-29895495-k5x2m" podUID="e3a2676b-0fd6-4c39-9ed7-d468706aa9e9"
Nov 03 18:15:05.740501 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:05.740480    5315 scope.go:122] "RemoveContainer" containerID="d8fa4764e246686dcba4005f14924b14ff01cc2bae3d23063475ee329e06b8db"
Nov 03 18:15:05.740833 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:15:05.740810    5315 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-80aca1-29895495-qvtbt_odh-ai-gateway-infra(26d8c8d7-89c0-47af-b90c-8b0677f71886)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-80aca1-29895495-qvtbt" podUID="26d8c8d7-89c0-47af-b90c-8b0677f71886"
Nov 03 18:15:05.742191 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:05.742149    5315 scope.go:122] "RemoveContainer" containerID="641a9509da3ff55fadbd9e96744036800cd32b606e9ad3de2ee1782a9abb0e8f"
Nov 03 18:15:05.742477 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:15:05.742452    5315 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895495-jgzv4_odh-ai-gateway-infra(e31a2c86-cff2-49ed-8f40-4020dfef4808)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-jgzv4" podUID="e31a2c86-cff2-49ed-8f40-4020dfef4808"
Nov 03 18:15:05.743740 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:05.743725    5315 scope.go:122] "RemoveContainer" containerID="6732a4b4ec6cd747d6a75f3808ff1431e120d5c9f95c0cebf37f4b0903d8e1c9"
Nov 03 18:15:05.743980 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:15:05.743957    5315 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-a86228-29895495-r68gb_odh-ai-gateway-infra(5d04a4ae-44e9-4f14-ada5-855c1efe7539)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895495-r68gb" podUID="5d04a4ae-44e9-4f14-ada5-855c1efe7539"
Nov 03 18:15:05.745289 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:05.745267    5315 scope.go:122] "RemoveContainer" containerID="821b86c994e38da0552da259b84afa48193aa291cc841f32c7e6bdd29b73d9b6"
Nov 03 18:15:05.745528 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:15:05.745505    5315 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-7a7c76-29895495-lhj7q_odh-ai-gateway-infra(8d8504d9-e3e9-4542-8e76-3dbd404ed89f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7a7c76-29895495-lhj7q" podUID="8d8504d9-e3e9-4542-8e76-3dbd404ed89f"
Nov 03 18:15:05.746764 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:05.746746    5315 scope.go:122] "RemoveContainer" containerID="3ce363f886ede99f0c20907b1c1c353410392b5fa53d5f07edc13c66d405a918"
Nov 03 18:15:05.746984 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:15:05.746965    5315 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-f79c02-29895495-shccb_odh-ai-gateway-infra(54bd478e-0685-4ec9-8aa0-5c3bd73c56c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895495-shccb" podUID="54bd478e-0685-4ec9-8aa0-5c3bd73c56c2"
Nov 03 18:15:05.925519 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:05.925486    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a86228-7nnxk"
Nov 03 18:15:05.985465 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:05.985413    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51-kube-api-access-66mfm\" (UniqueName: \"kubernetes.io/projected/658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51-kube-api-access-66mfm\") pod \"658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51\" (UID: \"658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51\") "
Nov 03 18:15:05.985693 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:05.985643    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51-maas-api-xks-ca\") pod \"658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51\" (UID: \"658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51\") "
Nov 03 18:15:05.985693 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:05.985682    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51-maas-api-openshift-ca\") pod \"658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51\" (UID: \"658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51\") "
Nov 03 18:15:05.986038 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:05.986012    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51-maas-api-openshift-ca" pod "658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51" (UID: "658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:05.986267 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:05.986248    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:05.987684 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:05.987657    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51-maas-api-xks-ca" pod "658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51" (UID: "658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:15:05.987796 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:05.987776    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51-kube-api-access-66mfm" pod "658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51" (UID: "658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51"). InnerVolumeSpecName "kube-api-access-66mfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:06.087202 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.087163    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66mfm\" (UniqueName: \"kubernetes.io/projected/658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51-kube-api-access-66mfm\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:06.087202 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.087196    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:06.265714 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.265681    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d"
Nov 03 18:15:06.297051 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.297013    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7338-ea4257"]
Nov 03 18:15:06.297626 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.297601    5315 cpu_manager.go:414] "containerMap: removing container" podUID="658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51" containerName="revoke-keys"
Nov 03 18:15:06.297671 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.297628    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51" containerName="revoke-keys"
Nov 03 18:15:06.297704 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.297676    5315 cpu_manager.go:414] "containerMap: removing container" podUID="ae6e0c1a-a26e-4b44-8e94-540944db8bd8" containerName="istio-proxy"
Nov 03 18:15:06.297704 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.297682    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae6e0c1a-a26e-4b44-8e94-540944db8bd8" containerName="istio-proxy"
Nov 03 18:15:06.297704 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.297692    5315 cpu_manager.go:414] "containerMap: removing container" podUID="ae6e0c1a-a26e-4b44-8e94-540944db8bd8" containerName="istio-proxy"
Nov 03 18:15:06.297704 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.297697    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae6e0c1a-a26e-4b44-8e94-540944db8bd8" containerName="istio-proxy"
Nov 03 18:15:06.297841 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.297785    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae6e0c1a-a26e-4b44-8e94-540944db8bd8" containerName="istio-proxy"
Nov 03 18:15:06.297841 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.297794    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae6e0c1a-a26e-4b44-8e94-540944db8bd8" containerName="istio-proxy"
Nov 03 18:15:06.297841 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.297804    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51" containerName="revoke-keys"
Nov 03 18:15:06.301387 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.301365    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7338-ea4257"
Nov 03 18:15:06.349987 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.349953    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-447v9\""
Nov 03 18:15:06.358738 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.358707    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7338-ea4257"]
Nov 03 18:15:06.398460 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.398418    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-istio-token\" (UniqueName: \"kubernetes.io/projected/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-istio-token\") pod \"ae6e0c1a-a26e-4b44-8e94-540944db8bd8\" (UID: \"ae6e0c1a-a26e-4b44-8e94-540944db8bd8\") "
Nov 03 18:15:06.398703 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.398473    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-workload-certs\") pod \"ae6e0c1a-a26e-4b44-8e94-540944db8bd8\" (UID: \"ae6e0c1a-a26e-4b44-8e94-540944db8bd8\") "
Nov 03 18:15:06.398703 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.398509    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-istio-envoy\") pod \"ae6e0c1a-a26e-4b44-8e94-540944db8bd8\" (UID: \"ae6e0c1a-a26e-4b44-8e94-540944db8bd8\") "
Nov 03 18:15:06.398946 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.398913    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-istio-data\") pod \"ae6e0c1a-a26e-4b44-8e94-540944db8bd8\" (UID: \"ae6e0c1a-a26e-4b44-8e94-540944db8bd8\") "
Nov 03 18:15:06.399258 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.399227    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-istio-data" pod "ae6e0c1a-a26e-4b44-8e94-540944db8bd8" (UID: "ae6e0c1a-a26e-4b44-8e94-540944db8bd8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:06.399351 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.399260    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-workload-certs" pod "ae6e0c1a-a26e-4b44-8e94-540944db8bd8" (UID: "ae6e0c1a-a26e-4b44-8e94-540944db8bd8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:06.399418 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.399371    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-credential-socket\") pod \"ae6e0c1a-a26e-4b44-8e94-540944db8bd8\" (UID: \"ae6e0c1a-a26e-4b44-8e94-540944db8bd8\") "
Nov 03 18:15:06.399470 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.399425    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-workload-socket\") pod \"ae6e0c1a-a26e-4b44-8e94-540944db8bd8\" (UID: \"ae6e0c1a-a26e-4b44-8e94-540944db8bd8\") "
Nov 03 18:15:06.399525 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.399492    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-istiod-ca-cert\") pod \"ae6e0c1a-a26e-4b44-8e94-540944db8bd8\" (UID: \"ae6e0c1a-a26e-4b44-8e94-540944db8bd8\") "
Nov 03 18:15:06.399763 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.399725    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-kube-api-access-v8hvh\" (UniqueName: \"kubernetes.io/projected/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-kube-api-access-v8hvh\") pod \"ae6e0c1a-a26e-4b44-8e94-540944db8bd8\" (UID: \"ae6e0c1a-a26e-4b44-8e94-540944db8bd8\") "
Nov 03 18:15:06.399848 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.399797    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-istio-podinfo\") pod \"ae6e0c1a-a26e-4b44-8e94-540944db8bd8\" (UID: \"ae6e0c1a-a26e-4b44-8e94-540944db8bd8\") "
Nov 03 18:15:06.401078 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.401049    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsjzd\" (UniqueName: \"kubernetes.io/projected/1703c5b7-dba5-4255-92e4-7f84f1554985-kube-api-access-vsjzd\") pod \"test-curl-7338-ea4257\" (UID: \"1703c5b7-dba5-4255-92e4-7f84f1554985\") " pod="opendatahub/test-curl-7338-ea4257"
Nov 03 18:15:06.402270 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.402249    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:06.402441 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.402414    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:06.402441 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.401592    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-credential-socket" pod "ae6e0c1a-a26e-4b44-8e94-540944db8bd8" (UID: "ae6e0c1a-a26e-4b44-8e94-540944db8bd8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:06.402584 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.401697    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-workload-socket" pod "ae6e0c1a-a26e-4b44-8e94-540944db8bd8" (UID: "ae6e0c1a-a26e-4b44-8e94-540944db8bd8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:06.402661 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.402598    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-istio-token" pod "ae6e0c1a-a26e-4b44-8e94-540944db8bd8" (UID: "ae6e0c1a-a26e-4b44-8e94-540944db8bd8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:06.405092 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.405062    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-istiod-ca-cert" pod "ae6e0c1a-a26e-4b44-8e94-540944db8bd8" (UID: "ae6e0c1a-a26e-4b44-8e94-540944db8bd8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:06.405481 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.405459    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-istio-envoy" pod "ae6e0c1a-a26e-4b44-8e94-540944db8bd8" (UID: "ae6e0c1a-a26e-4b44-8e94-540944db8bd8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:06.405906 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.405863    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-istio-podinfo" pod "ae6e0c1a-a26e-4b44-8e94-540944db8bd8" (UID: "ae6e0c1a-a26e-4b44-8e94-540944db8bd8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:15:06.407439 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.407415    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-kube-api-access-v8hvh" pod "ae6e0c1a-a26e-4b44-8e94-540944db8bd8" (UID: "ae6e0c1a-a26e-4b44-8e94-540944db8bd8"). InnerVolumeSpecName "kube-api-access-v8hvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:06.474516 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.474479    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c"]
Nov 03 18:15:06.482015 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.481975    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr"]
Nov 03 18:15:06.503458 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.503375    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsjzd\" (UniqueName: \"kubernetes.io/projected/1703c5b7-dba5-4255-92e4-7f84f1554985-kube-api-access-vsjzd\") pod \"test-curl-7338-ea4257\" (UID: \"1703c5b7-dba5-4255-92e4-7f84f1554985\") " pod="opendatahub/test-curl-7338-ea4257"
Nov 03 18:15:06.503690 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.503626    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:06.503690 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.503651    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:06.503690 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.503666    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8hvh\" (UniqueName: \"kubernetes.io/projected/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-kube-api-access-v8hvh\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:06.503690 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.503681    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:06.503918 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.503694    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:06.503918 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.503707    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:06.503918 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.503720    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6e0c1a-a26e-4b44-8e94-540944db8bd8-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:06.560292 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.560258    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsjzd\" (UniqueName: \"kubernetes.io/projected/1703c5b7-dba5-4255-92e4-7f84f1554985-kube-api-access-vsjzd\") pod \"test-curl-7338-ea4257\" (UID: \"1703c5b7-dba5-4255-92e4-7f84f1554985\") " pod="opendatahub/test-curl-7338-ea4257"
Nov 03 18:15:06.614802 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.614763    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7338-ea4257"
Nov 03 18:15:06.762711 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.757230    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a86228-7nnxk"
Nov 03 18:15:06.762711 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.757228    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a86228-7nnxk" event={"ID":"658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51","Type":"ContainerDied","Data":"16e9dfe3f571d3f8c70b06a04b6d4bf4ab5eaf28b7b5fb4b8bcbd80ec3cd0fa0"}
Nov 03 18:15:06.762711 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.757391    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16e9dfe3f571d3f8c70b06a04b6d4bf4ab5eaf28b7b5fb4b8bcbd80ec3cd0fa0"
Nov 03 18:15:06.762711 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.759160    5315 generic.go:356] "Generic (PLEG): container finished" podID="ae6e0c1a-a26e-4b44-8e94-540944db8bd8" containerID="76fc11ded9455c2a81036b73e1425af8d992264ffeb97286ccc3e0acb07b726c" exitCode=0
Nov 03 18:15:06.762711 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.759282    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d" event={"ID":"ae6e0c1a-a26e-4b44-8e94-540944db8bd8","Type":"ContainerDied","Data":"76fc11ded9455c2a81036b73e1425af8d992264ffeb97286ccc3e0acb07b726c"}
Nov 03 18:15:06.762711 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.759315    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d" event={"ID":"ae6e0c1a-a26e-4b44-8e94-540944db8bd8","Type":"ContainerDied","Data":"9ecc1b9db41027fde2bf399d9305ac94dee59bc9c6089161f8ccc872050f4ad0"}
Nov 03 18:15:06.762711 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.759336    5315 scope.go:122] "RemoveContainer" containerID="76fc11ded9455c2a81036b73e1425af8d992264ffeb97286ccc3e0acb07b726c"
Nov 03 18:15:06.762711 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.759462    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d"
Nov 03 18:15:06.777334 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.777303    5315 scope.go:122] "RemoveContainer" containerID="15b420fad5c451d5c774e7dea258301c1281506c0f5e05beb989a445f9b6ca1f"
Nov 03 18:15:06.787183 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.787157    5315 scope.go:122] "RemoveContainer" containerID="76fc11ded9455c2a81036b73e1425af8d992264ffeb97286ccc3e0acb07b726c"
Nov 03 18:15:06.787488 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:15:06.787467    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76fc11ded9455c2a81036b73e1425af8d992264ffeb97286ccc3e0acb07b726c\": container with ID starting with 76fc11ded9455c2a81036b73e1425af8d992264ffeb97286ccc3e0acb07b726c not found: ID does not exist" containerID="76fc11ded9455c2a81036b73e1425af8d992264ffeb97286ccc3e0acb07b726c"
Nov 03 18:15:06.787544 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.787498    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76fc11ded9455c2a81036b73e1425af8d992264ffeb97286ccc3e0acb07b726c"} err="failed to get container status \"76fc11ded9455c2a81036b73e1425af8d992264ffeb97286ccc3e0acb07b726c\": rpc error: code = NotFound desc = could not find container \"76fc11ded9455c2a81036b73e1425af8d992264ffeb97286ccc3e0acb07b726c\": container with ID starting with 76fc11ded9455c2a81036b73e1425af8d992264ffeb97286ccc3e0acb07b726c not found: ID does not exist"
Nov 03 18:15:06.787544 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.787516    5315 scope.go:122] "RemoveContainer" containerID="15b420fad5c451d5c774e7dea258301c1281506c0f5e05beb989a445f9b6ca1f"
Nov 03 18:15:06.787773 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:15:06.787756    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15b420fad5c451d5c774e7dea258301c1281506c0f5e05beb989a445f9b6ca1f\": container with ID starting with 15b420fad5c451d5c774e7dea258301c1281506c0f5e05beb989a445f9b6ca1f not found: ID does not exist" containerID="15b420fad5c451d5c774e7dea258301c1281506c0f5e05beb989a445f9b6ca1f"
Nov 03 18:15:06.787820 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.787779    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15b420fad5c451d5c774e7dea258301c1281506c0f5e05beb989a445f9b6ca1f"} err="failed to get container status \"15b420fad5c451d5c774e7dea258301c1281506c0f5e05beb989a445f9b6ca1f\": rpc error: code = NotFound desc = could not find container \"15b420fad5c451d5c774e7dea258301c1281506c0f5e05beb989a445f9b6ca1f\": container with ID starting with 15b420fad5c451d5c774e7dea258301c1281506c0f5e05beb989a445f9b6ca1f not found: ID does not exist"
Nov 03 18:15:06.999053 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:06.999008    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7338-ea4257"]
Nov 03 18:15:07.005179 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:15:07.005146    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1703c5b7_dba5_4255_92e4_7f84f1554985.slice/crio-212daddeb58dde359ab930871b7ebe004df2d708d873477ea121bc791688071a WatchSource:0}: Error finding container 212daddeb58dde359ab930871b7ebe004df2d708d873477ea121bc791688071a: Status 404 returned error can't find the container with id 212daddeb58dde359ab930871b7ebe004df2d708d873477ea121bc791688071a
Nov 03 18:15:07.039659 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:07.039604    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d"]
Nov 03 18:15:07.043355 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:07.043328    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-260dd0-openshift-default-5ff4755dfc-z9c6d"]
Nov 03 18:15:07.110149 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:07.110115    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae6e0c1a-a26e-4b44-8e94-540944db8bd8" path="/var/lib/kubelet/pods/ae6e0c1a-a26e-4b44-8e94-540944db8bd8/volumes"
Nov 03 18:15:07.773071 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:07.773032    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7338-ea4257" event={"ID":"1703c5b7-dba5-4255-92e4-7f84f1554985","Type":"ContainerStarted","Data":"72a86c94a4e2001152461c6a7ca1d66dcd0b007d47489a49b294ccdc67e3d6f6"}
Nov 03 18:15:07.773640 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:07.773087    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7338-ea4257" event={"ID":"1703c5b7-dba5-4255-92e4-7f84f1554985","Type":"ContainerStarted","Data":"212daddeb58dde359ab930871b7ebe004df2d708d873477ea121bc791688071a"}
Nov 03 18:15:07.891631 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:07.890446    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7338-ea4257" podStartSLOduration=1.890431391 podStartE2EDuration="1.890431391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:15:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:15:07.889346684 +0000 UTC m=+1746.905544758" watchObservedRunningTime="2026-11-03 18:15:07.890431391 +0000 UTC m=+1746.906629459"
Nov 03 18:15:09.343122 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:09.342031    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7338-ea4257"]
Nov 03 18:15:09.346910 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:09.346879    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7338-ea4257"]
Nov 03 18:15:09.827416 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:09.827337    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7338-ea4257" podUID="1703c5b7-dba5-4255-92e4-7f84f1554985" containerName="test-curl-7338-ea4257" containerID="cri-o://72a86c94a4e2001152461c6a7ca1d66dcd0b007d47489a49b294ccdc67e3d6f6" gracePeriod=2
Nov 03 18:15:09.875448 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:15:09.875404    5315 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7338-ea4257\" is forbidden: User \"system:node:ip-10-0-128-204.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-204.us-west-2.compute.internal' and this object" podUID="1703c5b7-dba5-4255-92e4-7f84f1554985" pod="opendatahub/test-curl-7338-ea4257"
Nov 03 18:15:10.210360 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:10.210235    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1720c00c-xm2ll"]
Nov 03 18:15:10.212765 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:10.211499    5315 cpu_manager.go:414] "containerMap: removing container" podUID="1703c5b7-dba5-4255-92e4-7f84f1554985" containerName="test-curl-7338-ea4257"
Nov 03 18:15:10.212765 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:10.211527    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1703c5b7-dba5-4255-92e4-7f84f1554985" containerName="test-curl-7338-ea4257"
Nov 03 18:15:10.212765 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:10.211757    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="1703c5b7-dba5-4255-92e4-7f84f1554985" containerName="test-curl-7338-ea4257"
Nov 03 18:15:10.217976 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:10.217947    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1720c00c-xm2ll"
Nov 03 18:15:10.266473 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:10.266429    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:15:10.268286 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:10.268260    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22a6a997-394b-4c8b-9de9-171c4d22ced1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-1720c00c-xm2ll\" (UID: \"22a6a997-394b-4c8b-9de9-171c4d22ced1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1720c00c-xm2ll"
Nov 03 18:15:10.268431 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:10.268320    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p84gw\" (UniqueName: \"kubernetes.io/projected/22a6a997-394b-4c8b-9de9-171c4d22ced1-kube-api-access-p84gw\") pod \"maas-api-revoke-keys-e2e-mt-1720c00c-xm2ll\" (UID: \"22a6a997-394b-4c8b-9de9-171c4d22ced1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1720c00c-xm2ll"
Nov 03 18:15:10.268497 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:10.268443    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22a6a997-394b-4c8b-9de9-171c4d22ced1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-1720c00c-xm2ll\" (UID: \"22a6a997-394b-4c8b-9de9-171c4d22ced1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1720c00c-xm2ll"
Nov 03 18:15:10.315681 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:10.315647    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1720c00c-xm2ll"]
Nov 03 18:15:10.369787 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:10.369747    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22a6a997-394b-4c8b-9de9-171c4d22ced1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-1720c00c-xm2ll\" (UID: \"22a6a997-394b-4c8b-9de9-171c4d22ced1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1720c00c-xm2ll"
Nov 03 18:15:10.370284 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:10.369920    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p84gw\" (UniqueName: \"kubernetes.io/projected/22a6a997-394b-4c8b-9de9-171c4d22ced1-kube-api-access-p84gw\") pod \"maas-api-revoke-keys-e2e-mt-1720c00c-xm2ll\" (UID: \"22a6a997-394b-4c8b-9de9-171c4d22ced1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1720c00c-xm2ll"
Nov 03 18:15:10.370284 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:10.370014    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22a6a997-394b-4c8b-9de9-171c4d22ced1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-1720c00c-xm2ll\" (UID: \"22a6a997-394b-4c8b-9de9-171c4d22ced1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1720c00c-xm2ll"
Nov 03 18:15:10.370694 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:10.370667    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22a6a997-394b-4c8b-9de9-171c4d22ced1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-1720c00c-xm2ll\" (UID: \"22a6a997-394b-4c8b-9de9-171c4d22ced1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1720c00c-xm2ll"
Nov 03 18:15:10.371376 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:15:10.371352    5315 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7338-ea4257\" is forbidden: User \"system:node:ip-10-0-128-204.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-204.us-west-2.compute.internal' and this object" podUID="1703c5b7-dba5-4255-92e4-7f84f1554985" pod="opendatahub/test-curl-7338-ea4257"
Nov 03 18:15:10.372194 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:10.372162    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22a6a997-394b-4c8b-9de9-171c4d22ced1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-1720c00c-xm2ll\" (UID: \"22a6a997-394b-4c8b-9de9-171c4d22ced1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1720c00c-xm2ll"
Nov 03 18:15:10.429585 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:10.427720    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p84gw\" (UniqueName: \"kubernetes.io/projected/22a6a997-394b-4c8b-9de9-171c4d22ced1-kube-api-access-p84gw\") pod \"maas-api-revoke-keys-e2e-mt-1720c00c-xm2ll\" (UID: \"22a6a997-394b-4c8b-9de9-171c4d22ced1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1720c00c-xm2ll"
Nov 03 18:15:10.534861 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:10.534825    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1720c00c-xm2ll"
Nov 03 18:15:10.909958 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:10.909869    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1720c00c-xm2ll"]
Nov 03 18:15:10.915473 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:15:10.915441    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22a6a997_394b_4c8b_9de9_171c4d22ced1.slice/crio-1a9ba81585766c3ffa6cb4e0dc73cb9dc883c6a66a621de523733e10f918c298 WatchSource:0}: Error finding container 1a9ba81585766c3ffa6cb4e0dc73cb9dc883c6a66a621de523733e10f918c298: Status 404 returned error can't find the container with id 1a9ba81585766c3ffa6cb4e0dc73cb9dc883c6a66a621de523733e10f918c298
Nov 03 18:15:11.148371 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:15:11.148338    5315 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7338-ea4257\" is forbidden: User \"system:node:ip-10-0-128-204.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-204.us-west-2.compute.internal' and this object" podUID="1703c5b7-dba5-4255-92e4-7f84f1554985" pod="opendatahub/test-curl-7338-ea4257"
Nov 03 18:15:11.844036 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:11.844000    5315 generic.go:356] "Generic (PLEG): container finished" podID="22a6a997-394b-4c8b-9de9-171c4d22ced1" containerID="7b414d4fc8af67fe9c6688cd6fa371d101b514861b6370c6e3ab0aeceaf5c3c8" exitCode=0
Nov 03 18:15:11.844511 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:11.844076    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1720c00c-xm2ll" event={"ID":"22a6a997-394b-4c8b-9de9-171c4d22ced1","Type":"ContainerDied","Data":"7b414d4fc8af67fe9c6688cd6fa371d101b514861b6370c6e3ab0aeceaf5c3c8"}
Nov 03 18:15:11.844511 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:11.844162    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1720c00c-xm2ll" event={"ID":"22a6a997-394b-4c8b-9de9-171c4d22ced1","Type":"ContainerStarted","Data":"1a9ba81585766c3ffa6cb4e0dc73cb9dc883c6a66a621de523733e10f918c298"}
Nov 03 18:15:12.048347 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:12.048274    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7338-ea4257"
Nov 03 18:15:12.107402 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:12.107366    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1703c5b7-dba5-4255-92e4-7f84f1554985-kube-api-access-vsjzd\" (UniqueName: \"kubernetes.io/projected/1703c5b7-dba5-4255-92e4-7f84f1554985-kube-api-access-vsjzd\") pod \"1703c5b7-dba5-4255-92e4-7f84f1554985\" (UID: \"1703c5b7-dba5-4255-92e4-7f84f1554985\") "
Nov 03 18:15:12.109710 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:12.109681    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1703c5b7-dba5-4255-92e4-7f84f1554985-kube-api-access-vsjzd" pod "1703c5b7-dba5-4255-92e4-7f84f1554985" (UID: "1703c5b7-dba5-4255-92e4-7f84f1554985"). InnerVolumeSpecName "kube-api-access-vsjzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:12.168782 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:12.168746    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0278b261-8vrsm"]
Nov 03 18:15:12.173132 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:12.173108    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0278b261-8vrsm"
Nov 03 18:15:12.208981 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:12.208934    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03b0d677-1c10-4757-8079-f00a09d4831e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0278b261-8vrsm\" (UID: \"03b0d677-1c10-4757-8079-f00a09d4831e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0278b261-8vrsm"
Nov 03 18:15:12.209191 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:12.209145    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03b0d677-1c10-4757-8079-f00a09d4831e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0278b261-8vrsm\" (UID: \"03b0d677-1c10-4757-8079-f00a09d4831e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0278b261-8vrsm"
Nov 03 18:15:12.209304 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:12.209281    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmt6p\" (UniqueName: \"kubernetes.io/projected/03b0d677-1c10-4757-8079-f00a09d4831e-kube-api-access-kmt6p\") pod \"maas-api-revoke-sub-0278b261-8vrsm\" (UID: \"03b0d677-1c10-4757-8079-f00a09d4831e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0278b261-8vrsm"
Nov 03 18:15:12.209465 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:12.209450    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsjzd\" (UniqueName: \"kubernetes.io/projected/1703c5b7-dba5-4255-92e4-7f84f1554985-kube-api-access-vsjzd\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:12.233525 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:12.233483    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0278b261-8vrsm"]
Nov 03 18:15:12.310485 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:12.310386    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03b0d677-1c10-4757-8079-f00a09d4831e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0278b261-8vrsm\" (UID: \"03b0d677-1c10-4757-8079-f00a09d4831e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0278b261-8vrsm"
Nov 03 18:15:12.310699 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:12.310501    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03b0d677-1c10-4757-8079-f00a09d4831e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0278b261-8vrsm\" (UID: \"03b0d677-1c10-4757-8079-f00a09d4831e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0278b261-8vrsm"
Nov 03 18:15:12.310699 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:12.310548    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmt6p\" (UniqueName: \"kubernetes.io/projected/03b0d677-1c10-4757-8079-f00a09d4831e-kube-api-access-kmt6p\") pod \"maas-api-revoke-sub-0278b261-8vrsm\" (UID: \"03b0d677-1c10-4757-8079-f00a09d4831e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0278b261-8vrsm"
Nov 03 18:15:12.311351 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:12.311318    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03b0d677-1c10-4757-8079-f00a09d4831e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0278b261-8vrsm\" (UID: \"03b0d677-1c10-4757-8079-f00a09d4831e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0278b261-8vrsm"
Nov 03 18:15:12.313010 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:12.312990    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03b0d677-1c10-4757-8079-f00a09d4831e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0278b261-8vrsm\" (UID: \"03b0d677-1c10-4757-8079-f00a09d4831e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0278b261-8vrsm"
Nov 03 18:15:12.376302 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:12.376270    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmt6p\" (UniqueName: \"kubernetes.io/projected/03b0d677-1c10-4757-8079-f00a09d4831e-kube-api-access-kmt6p\") pod \"maas-api-revoke-sub-0278b261-8vrsm\" (UID: \"03b0d677-1c10-4757-8079-f00a09d4831e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0278b261-8vrsm"
Nov 03 18:15:12.487418 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:12.487378    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0278b261-8vrsm"
Nov 03 18:15:12.852643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:12.852564    5315 generic.go:356] "Generic (PLEG): container finished" podID="1703c5b7-dba5-4255-92e4-7f84f1554985" containerID="72a86c94a4e2001152461c6a7ca1d66dcd0b007d47489a49b294ccdc67e3d6f6" exitCode=137
Nov 03 18:15:12.853145 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:12.852657    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7338-ea4257"
Nov 03 18:15:12.853145 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:12.852669    5315 scope.go:122] "RemoveContainer" containerID="72a86c94a4e2001152461c6a7ca1d66dcd0b007d47489a49b294ccdc67e3d6f6"
Nov 03 18:15:12.866353 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:12.866325    5315 scope.go:122] "RemoveContainer" containerID="72a86c94a4e2001152461c6a7ca1d66dcd0b007d47489a49b294ccdc67e3d6f6"
Nov 03 18:15:12.866933 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:15:12.866905    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72a86c94a4e2001152461c6a7ca1d66dcd0b007d47489a49b294ccdc67e3d6f6\": container with ID starting with 72a86c94a4e2001152461c6a7ca1d66dcd0b007d47489a49b294ccdc67e3d6f6 not found: ID does not exist" containerID="72a86c94a4e2001152461c6a7ca1d66dcd0b007d47489a49b294ccdc67e3d6f6"
Nov 03 18:15:12.867020 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:12.866949    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72a86c94a4e2001152461c6a7ca1d66dcd0b007d47489a49b294ccdc67e3d6f6"} err="failed to get container status \"72a86c94a4e2001152461c6a7ca1d66dcd0b007d47489a49b294ccdc67e3d6f6\": rpc error: code = NotFound desc = could not find container \"72a86c94a4e2001152461c6a7ca1d66dcd0b007d47489a49b294ccdc67e3d6f6\": container with ID starting with 72a86c94a4e2001152461c6a7ca1d66dcd0b007d47489a49b294ccdc67e3d6f6 not found: ID does not exist"
Nov 03 18:15:12.926418 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:15:12.913927    5315 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7338-ea4257\" is forbidden: User \"system:node:ip-10-0-128-204.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-204.us-west-2.compute.internal' and this object" podUID="1703c5b7-dba5-4255-92e4-7f84f1554985" pod="opendatahub/test-curl-7338-ea4257"
Nov 03 18:15:13.072333 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:13.054041    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0278b261-8vrsm"]
Nov 03 18:15:13.120496 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:13.120465    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1703c5b7-dba5-4255-92e4-7f84f1554985" path="/var/lib/kubelet/pods/1703c5b7-dba5-4255-92e4-7f84f1554985/volumes"
Nov 03 18:15:13.204155 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:13.204124    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1720c00c-xm2ll"
Nov 03 18:15:13.305042 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:13.305009    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22a6a997-394b-4c8b-9de9-171c4d22ced1-kube-api-access-p84gw\" (UniqueName: \"kubernetes.io/projected/22a6a997-394b-4c8b-9de9-171c4d22ced1-kube-api-access-p84gw\") pod \"22a6a997-394b-4c8b-9de9-171c4d22ced1\" (UID: \"22a6a997-394b-4c8b-9de9-171c4d22ced1\") "
Nov 03 18:15:13.305042 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:13.305054    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22a6a997-394b-4c8b-9de9-171c4d22ced1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22a6a997-394b-4c8b-9de9-171c4d22ced1-maas-api-xks-ca\") pod \"22a6a997-394b-4c8b-9de9-171c4d22ced1\" (UID: \"22a6a997-394b-4c8b-9de9-171c4d22ced1\") "
Nov 03 18:15:13.305278 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:13.305096    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22a6a997-394b-4c8b-9de9-171c4d22ced1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22a6a997-394b-4c8b-9de9-171c4d22ced1-maas-api-openshift-ca\") pod \"22a6a997-394b-4c8b-9de9-171c4d22ced1\" (UID: \"22a6a997-394b-4c8b-9de9-171c4d22ced1\") "
Nov 03 18:15:13.305509 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:13.305476    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22a6a997-394b-4c8b-9de9-171c4d22ced1-maas-api-openshift-ca" pod "22a6a997-394b-4c8b-9de9-171c4d22ced1" (UID: "22a6a997-394b-4c8b-9de9-171c4d22ced1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:13.307225 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:13.307200    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22a6a997-394b-4c8b-9de9-171c4d22ced1-maas-api-xks-ca" pod "22a6a997-394b-4c8b-9de9-171c4d22ced1" (UID: "22a6a997-394b-4c8b-9de9-171c4d22ced1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:15:13.307317 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:13.307279    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22a6a997-394b-4c8b-9de9-171c4d22ced1-kube-api-access-p84gw" pod "22a6a997-394b-4c8b-9de9-171c4d22ced1" (UID: "22a6a997-394b-4c8b-9de9-171c4d22ced1"). InnerVolumeSpecName "kube-api-access-p84gw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:13.422653 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:13.420953    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p84gw\" (UniqueName: \"kubernetes.io/projected/22a6a997-394b-4c8b-9de9-171c4d22ced1-kube-api-access-p84gw\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:13.422653 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:13.421001    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22a6a997-394b-4c8b-9de9-171c4d22ced1-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:13.422653 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:13.421018    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22a6a997-394b-4c8b-9de9-171c4d22ced1-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:13.742926 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:13.742887    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2853269b-rtts6"]
Nov 03 18:15:13.743794 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:13.743768    5315 cpu_manager.go:414] "containerMap: removing container" podUID="22a6a997-394b-4c8b-9de9-171c4d22ced1" containerName="revoke-keys"
Nov 03 18:15:13.743794 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:13.743792    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22a6a997-394b-4c8b-9de9-171c4d22ced1" containerName="revoke-keys"
Nov 03 18:15:13.744019 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:13.744004    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="22a6a997-394b-4c8b-9de9-171c4d22ced1" containerName="revoke-keys"
Nov 03 18:15:13.747556 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:13.747531    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2853269b-rtts6"
Nov 03 18:15:13.808925 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:13.808889    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2853269b-rtts6"]
Nov 03 18:15:13.830717 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:13.830681    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48396164-efa9-490e-9e12-13709493bd43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2853269b-rtts6\" (UID: \"48396164-efa9-490e-9e12-13709493bd43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2853269b-rtts6"
Nov 03 18:15:13.830886 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:13.830727    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtnxw\" (UniqueName: \"kubernetes.io/projected/48396164-efa9-490e-9e12-13709493bd43-kube-api-access-wtnxw\") pod \"maas-api-revoke-sub-2853269b-rtts6\" (UID: \"48396164-efa9-490e-9e12-13709493bd43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2853269b-rtts6"
Nov 03 18:15:13.831091 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:13.831058    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48396164-efa9-490e-9e12-13709493bd43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2853269b-rtts6\" (UID: \"48396164-efa9-490e-9e12-13709493bd43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2853269b-rtts6"
Nov 03 18:15:13.863039 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:13.862934    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0278b261-8vrsm" event={"ID":"03b0d677-1c10-4757-8079-f00a09d4831e","Type":"ContainerStarted","Data":"be6ab63213a6099f807323df01077cdaf4ca0d48196999f893b7511dc744b446"}
Nov 03 18:15:13.863039 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:13.863001    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0278b261-8vrsm" event={"ID":"03b0d677-1c10-4757-8079-f00a09d4831e","Type":"ContainerStarted","Data":"66f81dcc37b4a7cc5cc247c0aba9a7497251afc49c82a7944a4858b84e632cf4"}
Nov 03 18:15:13.864377 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:13.864344    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1720c00c-xm2ll"
Nov 03 18:15:13.864507 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:13.864348    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1720c00c-xm2ll" event={"ID":"22a6a997-394b-4c8b-9de9-171c4d22ced1","Type":"ContainerDied","Data":"1a9ba81585766c3ffa6cb4e0dc73cb9dc883c6a66a621de523733e10f918c298"}
Nov 03 18:15:13.864507 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:13.864474    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a9ba81585766c3ffa6cb4e0dc73cb9dc883c6a66a621de523733e10f918c298"
Nov 03 18:15:13.932698 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:13.932660    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48396164-efa9-490e-9e12-13709493bd43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2853269b-rtts6\" (UID: \"48396164-efa9-490e-9e12-13709493bd43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2853269b-rtts6"
Nov 03 18:15:13.932698 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:13.932702    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtnxw\" (UniqueName: \"kubernetes.io/projected/48396164-efa9-490e-9e12-13709493bd43-kube-api-access-wtnxw\") pod \"maas-api-revoke-sub-2853269b-rtts6\" (UID: \"48396164-efa9-490e-9e12-13709493bd43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2853269b-rtts6"
Nov 03 18:15:13.932980 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:13.932949    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48396164-efa9-490e-9e12-13709493bd43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2853269b-rtts6\" (UID: \"48396164-efa9-490e-9e12-13709493bd43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2853269b-rtts6"
Nov 03 18:15:13.933714 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:13.933684    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48396164-efa9-490e-9e12-13709493bd43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2853269b-rtts6\" (UID: \"48396164-efa9-490e-9e12-13709493bd43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2853269b-rtts6"
Nov 03 18:15:13.935069 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:13.935050    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48396164-efa9-490e-9e12-13709493bd43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2853269b-rtts6\" (UID: \"48396164-efa9-490e-9e12-13709493bd43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2853269b-rtts6"
Nov 03 18:15:13.980378 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:13.980308    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0278b261-8vrsm" podStartSLOduration=1.980291894 podStartE2EDuration="1.980291894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:15:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:15:13.976104214 +0000 UTC m=+1752.992302281" watchObservedRunningTime="2026-11-03 18:15:13.980291894 +0000 UTC m=+1752.996489962"
Nov 03 18:15:13.992450 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:13.992418    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtnxw\" (UniqueName: \"kubernetes.io/projected/48396164-efa9-490e-9e12-13709493bd43-kube-api-access-wtnxw\") pod \"maas-api-revoke-sub-2853269b-rtts6\" (UID: \"48396164-efa9-490e-9e12-13709493bd43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2853269b-rtts6"
Nov 03 18:15:14.061380 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:14.061342    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2853269b-rtts6"
Nov 03 18:15:14.097088 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:14.097060    5315 scope.go:122] "RemoveContainer" containerID="a6b7bcc438fc29b822204297cb9755cdaaf0cad240daf00031cacb69c32bde80"
Nov 03 18:15:14.479128 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:14.479091    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2853269b-rtts6"]
Nov 03 18:15:14.484759 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:15:14.484722    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48396164_efa9_490e_9e12_13709493bd43.slice/crio-3b51c982703cb146a236dc25fd3d81fca579b2f3db708498a9afcc8211723333 WatchSource:0}: Error finding container 3b51c982703cb146a236dc25fd3d81fca579b2f3db708498a9afcc8211723333: Status 404 returned error can't find the container with id 3b51c982703cb146a236dc25fd3d81fca579b2f3db708498a9afcc8211723333
Nov 03 18:15:14.503378 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:14.503349    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0adcbc-g486h"]
Nov 03 18:15:14.511756 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:14.511720    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0adcbc-g486h"]
Nov 03 18:15:14.679931 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:14.679843    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-a86228-f656db785-jcvfr"]
Nov 03 18:15:14.877149 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:14.877108    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c1b2d2-29895495-k5x2m" event={"ID":"e3a2676b-0fd6-4c39-9ed7-d468706aa9e9","Type":"ContainerStarted","Data":"0a6c720782689984d8d237123120916788a7fac514cf05f5d230b212d460cd58"}
Nov 03 18:15:14.879782 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:14.879747    5315 generic.go:356] "Generic (PLEG): container finished" podID="03b0d677-1c10-4757-8079-f00a09d4831e" containerID="be6ab63213a6099f807323df01077cdaf4ca0d48196999f893b7511dc744b446" exitCode=0
Nov 03 18:15:14.879917 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:14.879807    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0278b261-8vrsm" event={"ID":"03b0d677-1c10-4757-8079-f00a09d4831e","Type":"ContainerDied","Data":"be6ab63213a6099f807323df01077cdaf4ca0d48196999f893b7511dc744b446"}
Nov 03 18:15:14.881238 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:14.881214    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2853269b-rtts6" event={"ID":"48396164-efa9-490e-9e12-13709493bd43","Type":"ContainerStarted","Data":"3b51c982703cb146a236dc25fd3d81fca579b2f3db708498a9afcc8211723333"}
Nov 03 18:15:15.097129 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:15.097092    5315 scope.go:122] "RemoveContainer" containerID="3ce363f886ede99f0c20907b1c1c353410392b5fa53d5f07edc13c66d405a918"
Nov 03 18:15:15.105330 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:15.105304    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af71653e-b4ad-48a1-8ba9-bc8c1b10c44c" path="/var/lib/kubelet/pods/af71653e-b4ad-48a1-8ba9-bc8c1b10c44c/volumes"
Nov 03 18:15:15.890582 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:15.890543    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895495-shccb" event={"ID":"54bd478e-0685-4ec9-8aa0-5c3bd73c56c2","Type":"ContainerStarted","Data":"00733fd18a5fc690f3e550803e2616bf392d0e10347a1409de399a80a52c55e0"}
Nov 03 18:15:15.892711 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:15.892687    5315 generic.go:356] "Generic (PLEG): container finished" podID="e3a2676b-0fd6-4c39-9ed7-d468706aa9e9" containerID="0a6c720782689984d8d237123120916788a7fac514cf05f5d230b212d460cd58" exitCode=60
Nov 03 18:15:15.892867 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:15.892756    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c1b2d2-29895495-k5x2m" event={"ID":"e3a2676b-0fd6-4c39-9ed7-d468706aa9e9","Type":"ContainerDied","Data":"0a6c720782689984d8d237123120916788a7fac514cf05f5d230b212d460cd58"}
Nov 03 18:15:15.892867 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:15.892814    5315 scope.go:122] "RemoveContainer" containerID="a6b7bcc438fc29b822204297cb9755cdaaf0cad240daf00031cacb69c32bde80"
Nov 03 18:15:15.893208 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:15.893189    5315 scope.go:122] "RemoveContainer" containerID="0a6c720782689984d8d237123120916788a7fac514cf05f5d230b212d460cd58"
Nov 03 18:15:15.893581 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:15:15.893551    5315 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-c1b2d2-29895495-k5x2m_odh-ai-gateway-infra(e3a2676b-0fd6-4c39-9ed7-d468706aa9e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c1b2d2-29895495-k5x2m" podUID="e3a2676b-0fd6-4c39-9ed7-d468706aa9e9"
Nov 03 18:15:15.894548 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:15.894527    5315 generic.go:356] "Generic (PLEG): container finished" podID="48396164-efa9-490e-9e12-13709493bd43" containerID="a0e175c822312d8d7c282bd324796897a6deca6c690e94c82b0a4eb01ee5abfd" exitCode=0
Nov 03 18:15:15.894662 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:15.894638    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2853269b-rtts6" event={"ID":"48396164-efa9-490e-9e12-13709493bd43","Type":"ContainerDied","Data":"a0e175c822312d8d7c282bd324796897a6deca6c690e94c82b0a4eb01ee5abfd"}
Nov 03 18:15:16.009156 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:16.009117    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c1b2d2-29895495-k5x2m"]
Nov 03 18:15:16.098069 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:16.098038    5315 scope.go:122] "RemoveContainer" containerID="d8fa4764e246686dcba4005f14924b14ff01cc2bae3d23063475ee329e06b8db"
Nov 03 18:15:16.098255 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:16.098159    5315 scope.go:122] "RemoveContainer" containerID="821b86c994e38da0552da259b84afa48193aa291cc841f32c7e6bdd29b73d9b6"
Nov 03 18:15:16.126270 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:16.126235    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0278b261-8vrsm"
Nov 03 18:15:16.167084 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:16.167051    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03b0d677-1c10-4757-8079-f00a09d4831e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03b0d677-1c10-4757-8079-f00a09d4831e-maas-api-openshift-ca\") pod \"03b0d677-1c10-4757-8079-f00a09d4831e\" (UID: \"03b0d677-1c10-4757-8079-f00a09d4831e\") "
Nov 03 18:15:16.167255 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:16.167093    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03b0d677-1c10-4757-8079-f00a09d4831e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03b0d677-1c10-4757-8079-f00a09d4831e-maas-api-xks-ca\") pod \"03b0d677-1c10-4757-8079-f00a09d4831e\" (UID: \"03b0d677-1c10-4757-8079-f00a09d4831e\") "
Nov 03 18:15:16.167255 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:16.167165    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03b0d677-1c10-4757-8079-f00a09d4831e-kube-api-access-kmt6p\" (UniqueName: \"kubernetes.io/projected/03b0d677-1c10-4757-8079-f00a09d4831e-kube-api-access-kmt6p\") pod \"03b0d677-1c10-4757-8079-f00a09d4831e\" (UID: \"03b0d677-1c10-4757-8079-f00a09d4831e\") "
Nov 03 18:15:16.167437 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:16.167413    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03b0d677-1c10-4757-8079-f00a09d4831e-maas-api-openshift-ca" pod "03b0d677-1c10-4757-8079-f00a09d4831e" (UID: "03b0d677-1c10-4757-8079-f00a09d4831e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:16.168098 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:16.168059    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03b0d677-1c10-4757-8079-f00a09d4831e-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:16.169281 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:16.169253    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03b0d677-1c10-4757-8079-f00a09d4831e-maas-api-xks-ca" pod "03b0d677-1c10-4757-8079-f00a09d4831e" (UID: "03b0d677-1c10-4757-8079-f00a09d4831e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:15:16.169380 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:16.169359    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03b0d677-1c10-4757-8079-f00a09d4831e-kube-api-access-kmt6p" pod "03b0d677-1c10-4757-8079-f00a09d4831e" (UID: "03b0d677-1c10-4757-8079-f00a09d4831e"). InnerVolumeSpecName "kube-api-access-kmt6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:16.283646 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:16.281680    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03b0d677-1c10-4757-8079-f00a09d4831e-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:16.283646 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:16.281720    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmt6p\" (UniqueName: \"kubernetes.io/projected/03b0d677-1c10-4757-8079-f00a09d4831e-kube-api-access-kmt6p\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:16.906323 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:16.906289    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7a7c76-29895495-lhj7q" event={"ID":"8d8504d9-e3e9-4542-8e76-3dbd404ed89f","Type":"ContainerStarted","Data":"0b6f31d62b15fca0f3b0deafacbec9128fcc85da9917c8391fe435bbe819ca16"}
Nov 03 18:15:16.908389 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:16.908361    5315 generic.go:356] "Generic (PLEG): container finished" podID="54bd478e-0685-4ec9-8aa0-5c3bd73c56c2" containerID="00733fd18a5fc690f3e550803e2616bf392d0e10347a1409de399a80a52c55e0" exitCode=60
Nov 03 18:15:16.908542 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:16.908433    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895495-shccb" event={"ID":"54bd478e-0685-4ec9-8aa0-5c3bd73c56c2","Type":"ContainerDied","Data":"00733fd18a5fc690f3e550803e2616bf392d0e10347a1409de399a80a52c55e0"}
Nov 03 18:15:16.908542 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:16.908504    5315 scope.go:122] "RemoveContainer" containerID="3ce363f886ede99f0c20907b1c1c353410392b5fa53d5f07edc13c66d405a918"
Nov 03 18:15:16.908891 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:16.908870    5315 scope.go:122] "RemoveContainer" containerID="00733fd18a5fc690f3e550803e2616bf392d0e10347a1409de399a80a52c55e0"
Nov 03 18:15:16.909209 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:15:16.909182    5315 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-f79c02-29895495-shccb_odh-ai-gateway-infra(54bd478e-0685-4ec9-8aa0-5c3bd73c56c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895495-shccb" podUID="54bd478e-0685-4ec9-8aa0-5c3bd73c56c2"
Nov 03 18:15:16.913793 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:16.913770    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-80aca1-29895495-qvtbt" event={"ID":"26d8c8d7-89c0-47af-b90c-8b0677f71886","Type":"ContainerStarted","Data":"43ce9103e2b6b78beacea4a35770da72af78ea7168b3ff7e1f07545fc55836c8"}
Nov 03 18:15:16.915881 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:16.915711    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0278b261-8vrsm"
Nov 03 18:15:16.915881 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:16.915744    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0278b261-8vrsm" event={"ID":"03b0d677-1c10-4757-8079-f00a09d4831e","Type":"ContainerDied","Data":"66f81dcc37b4a7cc5cc247c0aba9a7497251afc49c82a7944a4858b84e632cf4"}
Nov 03 18:15:16.915881 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:16.915799    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66f81dcc37b4a7cc5cc247c0aba9a7497251afc49c82a7944a4858b84e632cf4"
Nov 03 18:15:17.067705 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.067667    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895495-shccb"]
Nov 03 18:15:17.098639 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.097582    5315 scope.go:122] "RemoveContainer" containerID="6732a4b4ec6cd747d6a75f3808ff1431e120d5c9f95c0cebf37f4b0903d8e1c9"
Nov 03 18:15:17.134841 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.134813    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c1b2d2-29895495-k5x2m"
Nov 03 18:15:17.188115 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.188076    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2853269b-rtts6"
Nov 03 18:15:17.221224 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.221189    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3a2676b-0fd6-4c39-9ed7-d468706aa9e9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3a2676b-0fd6-4c39-9ed7-d468706aa9e9-maas-api-service-ca\") pod \"e3a2676b-0fd6-4c39-9ed7-d468706aa9e9\" (UID: \"e3a2676b-0fd6-4c39-9ed7-d468706aa9e9\") "
Nov 03 18:15:17.221446 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.221310    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3a2676b-0fd6-4c39-9ed7-d468706aa9e9-kube-api-access-gbvzh\" (UniqueName: \"kubernetes.io/projected/e3a2676b-0fd6-4c39-9ed7-d468706aa9e9-kube-api-access-gbvzh\") pod \"e3a2676b-0fd6-4c39-9ed7-d468706aa9e9\" (UID: \"e3a2676b-0fd6-4c39-9ed7-d468706aa9e9\") "
Nov 03 18:15:17.226186 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.221585    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3a2676b-0fd6-4c39-9ed7-d468706aa9e9-maas-api-service-ca" pod "e3a2676b-0fd6-4c39-9ed7-d468706aa9e9" (UID: "e3a2676b-0fd6-4c39-9ed7-d468706aa9e9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:17.226186 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.222158    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3a2676b-0fd6-4c39-9ed7-d468706aa9e9-maas-api-service-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:17.227775 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.227738    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3a2676b-0fd6-4c39-9ed7-d468706aa9e9-kube-api-access-gbvzh" pod "e3a2676b-0fd6-4c39-9ed7-d468706aa9e9" (UID: "e3a2676b-0fd6-4c39-9ed7-d468706aa9e9"). InnerVolumeSpecName "kube-api-access-gbvzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:17.323689 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.323653    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48396164-efa9-490e-9e12-13709493bd43-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48396164-efa9-490e-9e12-13709493bd43-maas-api-xks-ca\") pod \"48396164-efa9-490e-9e12-13709493bd43\" (UID: \"48396164-efa9-490e-9e12-13709493bd43\") "
Nov 03 18:15:17.323899 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.323786    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48396164-efa9-490e-9e12-13709493bd43-kube-api-access-wtnxw\" (UniqueName: \"kubernetes.io/projected/48396164-efa9-490e-9e12-13709493bd43-kube-api-access-wtnxw\") pod \"48396164-efa9-490e-9e12-13709493bd43\" (UID: \"48396164-efa9-490e-9e12-13709493bd43\") "
Nov 03 18:15:17.323899 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.323828    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48396164-efa9-490e-9e12-13709493bd43-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48396164-efa9-490e-9e12-13709493bd43-maas-api-openshift-ca\") pod \"48396164-efa9-490e-9e12-13709493bd43\" (UID: \"48396164-efa9-490e-9e12-13709493bd43\") "
Nov 03 18:15:17.324196 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.324168    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48396164-efa9-490e-9e12-13709493bd43-maas-api-openshift-ca" pod "48396164-efa9-490e-9e12-13709493bd43" (UID: "48396164-efa9-490e-9e12-13709493bd43"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:17.324603 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.324578    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbvzh\" (UniqueName: \"kubernetes.io/projected/e3a2676b-0fd6-4c39-9ed7-d468706aa9e9-kube-api-access-gbvzh\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:17.325495 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.325475    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48396164-efa9-490e-9e12-13709493bd43-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:17.326566 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.326509    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48396164-efa9-490e-9e12-13709493bd43-maas-api-xks-ca" pod "48396164-efa9-490e-9e12-13709493bd43" (UID: "48396164-efa9-490e-9e12-13709493bd43"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:15:17.326755 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.326722    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48396164-efa9-490e-9e12-13709493bd43-kube-api-access-wtnxw" pod "48396164-efa9-490e-9e12-13709493bd43" (UID: "48396164-efa9-490e-9e12-13709493bd43"). InnerVolumeSpecName "kube-api-access-wtnxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:17.427412 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.427376    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtnxw\" (UniqueName: \"kubernetes.io/projected/48396164-efa9-490e-9e12-13709493bd43-kube-api-access-wtnxw\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:17.427412 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.427410    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48396164-efa9-490e-9e12-13709493bd43-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:17.527545 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.527510    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6"]
Nov 03 18:15:17.528296 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.528274    5315 cpu_manager.go:414] "containerMap: removing container" podUID="48396164-efa9-490e-9e12-13709493bd43" containerName="revoke-keys"
Nov 03 18:15:17.528296 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.528297    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48396164-efa9-490e-9e12-13709493bd43" containerName="revoke-keys"
Nov 03 18:15:17.528468 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.528317    5315 cpu_manager.go:414] "containerMap: removing container" podUID="e3a2676b-0fd6-4c39-9ed7-d468706aa9e9" containerName="cleanup"
Nov 03 18:15:17.528468 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.528325    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3a2676b-0fd6-4c39-9ed7-d468706aa9e9" containerName="cleanup"
Nov 03 18:15:17.528468 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.528338    5315 cpu_manager.go:414] "containerMap: removing container" podUID="e3a2676b-0fd6-4c39-9ed7-d468706aa9e9" containerName="cleanup"
Nov 03 18:15:17.528468 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.528343    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3a2676b-0fd6-4c39-9ed7-d468706aa9e9" containerName="cleanup"
Nov 03 18:15:17.528468 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.528349    5315 cpu_manager.go:414] "containerMap: removing container" podUID="03b0d677-1c10-4757-8079-f00a09d4831e" containerName="revoke-keys"
Nov 03 18:15:17.528468 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.528354    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03b0d677-1c10-4757-8079-f00a09d4831e" containerName="revoke-keys"
Nov 03 18:15:17.528702 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.528475    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="48396164-efa9-490e-9e12-13709493bd43" containerName="revoke-keys"
Nov 03 18:15:17.528702 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.528486    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3a2676b-0fd6-4c39-9ed7-d468706aa9e9" containerName="cleanup"
Nov 03 18:15:17.528702 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.528494    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3a2676b-0fd6-4c39-9ed7-d468706aa9e9" containerName="cleanup"
Nov 03 18:15:17.528702 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.528499    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="03b0d677-1c10-4757-8079-f00a09d4831e" containerName="revoke-keys"
Nov 03 18:15:17.531825 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.531805    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6"
Nov 03 18:15:17.541021 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.540980    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb"]
Nov 03 18:15:17.541692 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.541672    5315 cpu_manager.go:414] "containerMap: removing container" podUID="e3a2676b-0fd6-4c39-9ed7-d468706aa9e9" containerName="cleanup"
Nov 03 18:15:17.541692 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.541692    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3a2676b-0fd6-4c39-9ed7-d468706aa9e9" containerName="cleanup"
Nov 03 18:15:17.541874 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.541860    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3a2676b-0fd6-4c39-9ed7-d468706aa9e9" containerName="cleanup"
Nov 03 18:15:17.546638 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.546601    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb"
Nov 03 18:15:17.595957 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.595866    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6"]
Nov 03 18:15:17.610095 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.610062    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb"]
Nov 03 18:15:17.629710 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.629671    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b0b7de-c4ed-4997-ba30-72f4d501786b-workload-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb\" (UID: \"14b0b7de-c4ed-4997-ba30-72f4d501786b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb"
Nov 03 18:15:17.629934 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.629728    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b0b7de-c4ed-4997-ba30-72f4d501786b-credential-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb\" (UID: \"14b0b7de-c4ed-4997-ba30-72f4d501786b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb"
Nov 03 18:15:17.629934 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.629852    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14b0b7de-c4ed-4997-ba30-72f4d501786b-istio-data\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb\" (UID: \"14b0b7de-c4ed-4997-ba30-72f4d501786b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb"
Nov 03 18:15:17.629934 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.629897    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14b0b7de-c4ed-4997-ba30-72f4d501786b-istio-token\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb\" (UID: \"14b0b7de-c4ed-4997-ba30-72f4d501786b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb"
Nov 03 18:15:17.629934 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.629920    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89bf9e11-1400-4a4a-b474-f23baf5d0da5-workload-certs\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6\" (UID: \"89bf9e11-1400-4a4a-b474-f23baf5d0da5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6"
Nov 03 18:15:17.630137 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.630030    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89bf9e11-1400-4a4a-b474-f23baf5d0da5-istio-podinfo\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6\" (UID: \"89bf9e11-1400-4a4a-b474-f23baf5d0da5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6"
Nov 03 18:15:17.630137 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.630088    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89bf9e11-1400-4a4a-b474-f23baf5d0da5-istio-envoy\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6\" (UID: \"89bf9e11-1400-4a4a-b474-f23baf5d0da5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6"
Nov 03 18:15:17.630137 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.630107    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89bf9e11-1400-4a4a-b474-f23baf5d0da5-istiod-ca-cert\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6\" (UID: \"89bf9e11-1400-4a4a-b474-f23baf5d0da5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6"
Nov 03 18:15:17.630238 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.630143    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5mv7\" (UniqueName: \"kubernetes.io/projected/89bf9e11-1400-4a4a-b474-f23baf5d0da5-kube-api-access-h5mv7\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6\" (UID: \"89bf9e11-1400-4a4a-b474-f23baf5d0da5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6"
Nov 03 18:15:17.630238 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.630220    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14b0b7de-c4ed-4997-ba30-72f4d501786b-istio-envoy\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb\" (UID: \"14b0b7de-c4ed-4997-ba30-72f4d501786b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb"
Nov 03 18:15:17.630298 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.630244    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89bf9e11-1400-4a4a-b474-f23baf5d0da5-istio-data\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6\" (UID: \"89bf9e11-1400-4a4a-b474-f23baf5d0da5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6"
Nov 03 18:15:17.630298 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.630272    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89bf9e11-1400-4a4a-b474-f23baf5d0da5-istio-token\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6\" (UID: \"89bf9e11-1400-4a4a-b474-f23baf5d0da5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6"
Nov 03 18:15:17.630388 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.630378    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14b0b7de-c4ed-4997-ba30-72f4d501786b-istiod-ca-cert\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb\" (UID: \"14b0b7de-c4ed-4997-ba30-72f4d501786b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb"
Nov 03 18:15:17.630435 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.630412    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpddv\" (UniqueName: \"kubernetes.io/projected/14b0b7de-c4ed-4997-ba30-72f4d501786b-kube-api-access-cpddv\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb\" (UID: \"14b0b7de-c4ed-4997-ba30-72f4d501786b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb"
Nov 03 18:15:17.630487 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.630434    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14b0b7de-c4ed-4997-ba30-72f4d501786b-istio-podinfo\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb\" (UID: \"14b0b7de-c4ed-4997-ba30-72f4d501786b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb"
Nov 03 18:15:17.630487 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.630456    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89bf9e11-1400-4a4a-b474-f23baf5d0da5-credential-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6\" (UID: \"89bf9e11-1400-4a4a-b474-f23baf5d0da5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6"
Nov 03 18:15:17.630660 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.630602    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89bf9e11-1400-4a4a-b474-f23baf5d0da5-workload-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6\" (UID: \"89bf9e11-1400-4a4a-b474-f23baf5d0da5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6"
Nov 03 18:15:17.630745 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.630708    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14b0b7de-c4ed-4997-ba30-72f4d501786b-workload-certs\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb\" (UID: \"14b0b7de-c4ed-4997-ba30-72f4d501786b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb"
Nov 03 18:15:17.731843 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.731807    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14b0b7de-c4ed-4997-ba30-72f4d501786b-istiod-ca-cert\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb\" (UID: \"14b0b7de-c4ed-4997-ba30-72f4d501786b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb"
Nov 03 18:15:17.732053 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.731857    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpddv\" (UniqueName: \"kubernetes.io/projected/14b0b7de-c4ed-4997-ba30-72f4d501786b-kube-api-access-cpddv\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb\" (UID: \"14b0b7de-c4ed-4997-ba30-72f4d501786b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb"
Nov 03 18:15:17.732053 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.731880    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14b0b7de-c4ed-4997-ba30-72f4d501786b-istio-podinfo\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb\" (UID: \"14b0b7de-c4ed-4997-ba30-72f4d501786b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb"
Nov 03 18:15:17.732053 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.731906    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89bf9e11-1400-4a4a-b474-f23baf5d0da5-credential-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6\" (UID: \"89bf9e11-1400-4a4a-b474-f23baf5d0da5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6"
Nov 03 18:15:17.732053 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.731953    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89bf9e11-1400-4a4a-b474-f23baf5d0da5-workload-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6\" (UID: \"89bf9e11-1400-4a4a-b474-f23baf5d0da5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6"
Nov 03 18:15:17.732053 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.732002    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14b0b7de-c4ed-4997-ba30-72f4d501786b-workload-certs\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb\" (UID: \"14b0b7de-c4ed-4997-ba30-72f4d501786b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb"
Nov 03 18:15:17.732053 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.732038    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b0b7de-c4ed-4997-ba30-72f4d501786b-workload-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb\" (UID: \"14b0b7de-c4ed-4997-ba30-72f4d501786b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb"
Nov 03 18:15:17.732424 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.732157    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b0b7de-c4ed-4997-ba30-72f4d501786b-credential-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb\" (UID: \"14b0b7de-c4ed-4997-ba30-72f4d501786b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb"
Nov 03 18:15:17.732424 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.732228    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14b0b7de-c4ed-4997-ba30-72f4d501786b-istio-data\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb\" (UID: \"14b0b7de-c4ed-4997-ba30-72f4d501786b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb"
Nov 03 18:15:17.732424 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.732254    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14b0b7de-c4ed-4997-ba30-72f4d501786b-istio-token\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb\" (UID: \"14b0b7de-c4ed-4997-ba30-72f4d501786b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb"
Nov 03 18:15:17.732424 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.732278    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89bf9e11-1400-4a4a-b474-f23baf5d0da5-workload-certs\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6\" (UID: \"89bf9e11-1400-4a4a-b474-f23baf5d0da5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6"
Nov 03 18:15:17.732424 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.732332    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89bf9e11-1400-4a4a-b474-f23baf5d0da5-credential-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6\" (UID: \"89bf9e11-1400-4a4a-b474-f23baf5d0da5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6"
Nov 03 18:15:17.732424 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.732344    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89bf9e11-1400-4a4a-b474-f23baf5d0da5-istio-podinfo\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6\" (UID: \"89bf9e11-1400-4a4a-b474-f23baf5d0da5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6"
Nov 03 18:15:17.732830 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.732433    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89bf9e11-1400-4a4a-b474-f23baf5d0da5-istio-envoy\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6\" (UID: \"89bf9e11-1400-4a4a-b474-f23baf5d0da5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6"
Nov 03 18:15:17.732830 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.732460    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89bf9e11-1400-4a4a-b474-f23baf5d0da5-istiod-ca-cert\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6\" (UID: \"89bf9e11-1400-4a4a-b474-f23baf5d0da5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6"
Nov 03 18:15:17.732830 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.732498    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5mv7\" (UniqueName: \"kubernetes.io/projected/89bf9e11-1400-4a4a-b474-f23baf5d0da5-kube-api-access-h5mv7\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6\" (UID: \"89bf9e11-1400-4a4a-b474-f23baf5d0da5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6"
Nov 03 18:15:17.732830 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.732557    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14b0b7de-c4ed-4997-ba30-72f4d501786b-istio-envoy\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb\" (UID: \"14b0b7de-c4ed-4997-ba30-72f4d501786b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb"
Nov 03 18:15:17.732830 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.732593    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89bf9e11-1400-4a4a-b474-f23baf5d0da5-istio-data\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6\" (UID: \"89bf9e11-1400-4a4a-b474-f23baf5d0da5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6"
Nov 03 18:15:17.732830 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.732642    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14b0b7de-c4ed-4997-ba30-72f4d501786b-istiod-ca-cert\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb\" (UID: \"14b0b7de-c4ed-4997-ba30-72f4d501786b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb"
Nov 03 18:15:17.732830 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.732655    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14b0b7de-c4ed-4997-ba30-72f4d501786b-workload-certs\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb\" (UID: \"14b0b7de-c4ed-4997-ba30-72f4d501786b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb"
Nov 03 18:15:17.732830 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.732665    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89bf9e11-1400-4a4a-b474-f23baf5d0da5-istio-token\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6\" (UID: \"89bf9e11-1400-4a4a-b474-f23baf5d0da5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6"
Nov 03 18:15:17.732830 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.732670    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89bf9e11-1400-4a4a-b474-f23baf5d0da5-workload-certs\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6\" (UID: \"89bf9e11-1400-4a4a-b474-f23baf5d0da5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6"
Nov 03 18:15:17.732830 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.732770    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b0b7de-c4ed-4997-ba30-72f4d501786b-credential-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb\" (UID: \"14b0b7de-c4ed-4997-ba30-72f4d501786b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb"
Nov 03 18:15:17.733434 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.732856    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14b0b7de-c4ed-4997-ba30-72f4d501786b-istio-data\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb\" (UID: \"14b0b7de-c4ed-4997-ba30-72f4d501786b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb"
Nov 03 18:15:17.733434 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.732866    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b0b7de-c4ed-4997-ba30-72f4d501786b-workload-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb\" (UID: \"14b0b7de-c4ed-4997-ba30-72f4d501786b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb"
Nov 03 18:15:17.733434 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.732860    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89bf9e11-1400-4a4a-b474-f23baf5d0da5-workload-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6\" (UID: \"89bf9e11-1400-4a4a-b474-f23baf5d0da5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6"
Nov 03 18:15:17.733643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.733469    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89bf9e11-1400-4a4a-b474-f23baf5d0da5-istio-data\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6\" (UID: \"89bf9e11-1400-4a4a-b474-f23baf5d0da5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6"
Nov 03 18:15:17.733643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.733559    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89bf9e11-1400-4a4a-b474-f23baf5d0da5-istiod-ca-cert\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6\" (UID: \"89bf9e11-1400-4a4a-b474-f23baf5d0da5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6"
Nov 03 18:15:17.734912 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.734885    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14b0b7de-c4ed-4997-ba30-72f4d501786b-istio-podinfo\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb\" (UID: \"14b0b7de-c4ed-4997-ba30-72f4d501786b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb"
Nov 03 18:15:17.735570 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.735548    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89bf9e11-1400-4a4a-b474-f23baf5d0da5-istio-podinfo\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6\" (UID: \"89bf9e11-1400-4a4a-b474-f23baf5d0da5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6"
Nov 03 18:15:17.735666 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.735601    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89bf9e11-1400-4a4a-b474-f23baf5d0da5-istio-envoy\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6\" (UID: \"89bf9e11-1400-4a4a-b474-f23baf5d0da5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6"
Nov 03 18:15:17.735710 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.735602    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14b0b7de-c4ed-4997-ba30-72f4d501786b-istio-envoy\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb\" (UID: \"14b0b7de-c4ed-4997-ba30-72f4d501786b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb"
Nov 03 18:15:17.789361 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.789321    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14b0b7de-c4ed-4997-ba30-72f4d501786b-istio-token\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb\" (UID: \"14b0b7de-c4ed-4997-ba30-72f4d501786b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb"
Nov 03 18:15:17.789528 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.789392    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpddv\" (UniqueName: \"kubernetes.io/projected/14b0b7de-c4ed-4997-ba30-72f4d501786b-kube-api-access-cpddv\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb\" (UID: \"14b0b7de-c4ed-4997-ba30-72f4d501786b\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb"
Nov 03 18:15:17.789528 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.789438    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5mv7\" (UniqueName: \"kubernetes.io/projected/89bf9e11-1400-4a4a-b474-f23baf5d0da5-kube-api-access-h5mv7\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6\" (UID: \"89bf9e11-1400-4a4a-b474-f23baf5d0da5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6"
Nov 03 18:15:17.789528 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.789484    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89bf9e11-1400-4a4a-b474-f23baf5d0da5-istio-token\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6\" (UID: \"89bf9e11-1400-4a4a-b474-f23baf5d0da5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6"
Nov 03 18:15:17.848036 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.847948    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6"
Nov 03 18:15:17.861730 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.861700    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb"
Nov 03 18:15:17.941919 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.941880    5315 generic.go:356] "Generic (PLEG): container finished" podID="26d8c8d7-89c0-47af-b90c-8b0677f71886" containerID="43ce9103e2b6b78beacea4a35770da72af78ea7168b3ff7e1f07545fc55836c8" exitCode=60
Nov 03 18:15:17.942437 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.942048    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-80aca1-29895495-qvtbt" event={"ID":"26d8c8d7-89c0-47af-b90c-8b0677f71886","Type":"ContainerDied","Data":"43ce9103e2b6b78beacea4a35770da72af78ea7168b3ff7e1f07545fc55836c8"}
Nov 03 18:15:17.942437 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.942111    5315 scope.go:122] "RemoveContainer" containerID="d8fa4764e246686dcba4005f14924b14ff01cc2bae3d23063475ee329e06b8db"
Nov 03 18:15:17.942683 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.942661    5315 scope.go:122] "RemoveContainer" containerID="43ce9103e2b6b78beacea4a35770da72af78ea7168b3ff7e1f07545fc55836c8"
Nov 03 18:15:17.943110 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:15:17.943080    5315 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-80aca1-29895495-qvtbt_odh-ai-gateway-infra(26d8c8d7-89c0-47af-b90c-8b0677f71886)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-80aca1-29895495-qvtbt" podUID="26d8c8d7-89c0-47af-b90c-8b0677f71886"
Nov 03 18:15:17.948702 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.948660    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895495-r68gb" event={"ID":"5d04a4ae-44e9-4f14-ada5-855c1efe7539","Type":"ContainerStarted","Data":"a3b565e94bf5d4ff193d7a489145bf6edb645d6caabe8eefcfca68c307609b35"}
Nov 03 18:15:17.950396 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.950372    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2853269b-rtts6" event={"ID":"48396164-efa9-490e-9e12-13709493bd43","Type":"ContainerDied","Data":"3b51c982703cb146a236dc25fd3d81fca579b2f3db708498a9afcc8211723333"}
Nov 03 18:15:17.950512 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.950420    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b51c982703cb146a236dc25fd3d81fca579b2f3db708498a9afcc8211723333"
Nov 03 18:15:17.950512 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.950480    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2853269b-rtts6"
Nov 03 18:15:17.954281 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.954250    5315 generic.go:356] "Generic (PLEG): container finished" podID="8d8504d9-e3e9-4542-8e76-3dbd404ed89f" containerID="0b6f31d62b15fca0f3b0deafacbec9128fcc85da9917c8391fe435bbe819ca16" exitCode=60
Nov 03 18:15:17.954399 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.954336    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7a7c76-29895495-lhj7q" event={"ID":"8d8504d9-e3e9-4542-8e76-3dbd404ed89f","Type":"ContainerDied","Data":"0b6f31d62b15fca0f3b0deafacbec9128fcc85da9917c8391fe435bbe819ca16"}
Nov 03 18:15:17.955076 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.954797    5315 scope.go:122] "RemoveContainer" containerID="0b6f31d62b15fca0f3b0deafacbec9128fcc85da9917c8391fe435bbe819ca16"
Nov 03 18:15:17.955236 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:15:17.955168    5315 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-7a7c76-29895495-lhj7q_odh-ai-gateway-infra(8d8504d9-e3e9-4542-8e76-3dbd404ed89f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7a7c76-29895495-lhj7q" podUID="8d8504d9-e3e9-4542-8e76-3dbd404ed89f"
Nov 03 18:15:17.962143 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.962118    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c1b2d2-29895495-k5x2m" event={"ID":"e3a2676b-0fd6-4c39-9ed7-d468706aa9e9","Type":"ContainerDied","Data":"2ddaba5be9bd7d5a5eb0006478482141e992be06542ec1bfd35b1d975bdec356"}
Nov 03 18:15:17.962262 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.962216    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c1b2d2-29895495-k5x2m"
Nov 03 18:15:17.970776 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:17.970752    5315 scope.go:122] "RemoveContainer" containerID="821b86c994e38da0552da259b84afa48193aa291cc841f32c7e6bdd29b73d9b6"
Nov 03 18:15:18.009066 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:18.009039    5315 scope.go:122] "RemoveContainer" containerID="0a6c720782689984d8d237123120916788a7fac514cf05f5d230b212d460cd58"
Nov 03 18:15:18.077738 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:18.077702    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7a7c76-29895495-lhj7q"]
Nov 03 18:15:18.097801 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:18.097766    5315 scope.go:122] "RemoveContainer" containerID="641a9509da3ff55fadbd9e96744036800cd32b606e9ad3de2ee1782a9abb0e8f"
Nov 03 18:15:18.187845 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:18.187810    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-80aca1-29895495-qvtbt"]
Nov 03 18:15:18.199671 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:18.199646    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895495-shccb"
Nov 03 18:15:18.305252 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:18.305197    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6"]
Nov 03 18:15:18.307913 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:15:18.307884    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89bf9e11_1400_4a4a_b474_f23baf5d0da5.slice/crio-3218dcdb99b491de999a1748c5ad6e6b1e8c087c8b8a370d1d59f82ed7b151eb WatchSource:0}: Error finding container 3218dcdb99b491de999a1748c5ad6e6b1e8c087c8b8a370d1d59f82ed7b151eb: Status 404 returned error can't find the container with id 3218dcdb99b491de999a1748c5ad6e6b1e8c087c8b8a370d1d59f82ed7b151eb
Nov 03 18:15:18.309266 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:18.309243    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb"]
Nov 03 18:15:18.313046 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:18.313007    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:15:18.313225 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:18.313206    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:15:18.313356 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:18.313341    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:15:18.317876 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:18.317848    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:15:18.317960 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:18.317907    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:15:18.317960 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:18.317938    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:15:18.342854 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:18.342829    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54bd478e-0685-4ec9-8aa0-5c3bd73c56c2-kube-api-access-sswlc\" (UniqueName: \"kubernetes.io/projected/54bd478e-0685-4ec9-8aa0-5c3bd73c56c2-kube-api-access-sswlc\") pod \"54bd478e-0685-4ec9-8aa0-5c3bd73c56c2\" (UID: \"54bd478e-0685-4ec9-8aa0-5c3bd73c56c2\") "
Nov 03 18:15:18.342978 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:18.342911    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54bd478e-0685-4ec9-8aa0-5c3bd73c56c2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54bd478e-0685-4ec9-8aa0-5c3bd73c56c2-maas-api-service-ca\") pod \"54bd478e-0685-4ec9-8aa0-5c3bd73c56c2\" (UID: \"54bd478e-0685-4ec9-8aa0-5c3bd73c56c2\") "
Nov 03 18:15:18.343300 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:18.343276    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54bd478e-0685-4ec9-8aa0-5c3bd73c56c2-maas-api-service-ca" pod "54bd478e-0685-4ec9-8aa0-5c3bd73c56c2" (UID: "54bd478e-0685-4ec9-8aa0-5c3bd73c56c2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:18.343546 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:18.343523    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54bd478e-0685-4ec9-8aa0-5c3bd73c56c2-maas-api-service-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:18.345362 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:18.345340    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54bd478e-0685-4ec9-8aa0-5c3bd73c56c2-kube-api-access-sswlc" pod "54bd478e-0685-4ec9-8aa0-5c3bd73c56c2" (UID: "54bd478e-0685-4ec9-8aa0-5c3bd73c56c2"). InnerVolumeSpecName "kube-api-access-sswlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:18.444829 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:18.444796    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sswlc\" (UniqueName: \"kubernetes.io/projected/54bd478e-0685-4ec9-8aa0-5c3bd73c56c2-kube-api-access-sswlc\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:18.502178 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:18.502144    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c1b2d2-29895495-k5x2m"]
Nov 03 18:15:18.507926 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:18.507894    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c1b2d2-29895495-k5x2m"]
Nov 03 18:15:18.999636 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:18.998673    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6" event={"ID":"89bf9e11-1400-4a4a-b474-f23baf5d0da5","Type":"ContainerStarted","Data":"532430e4867c14237c848ad891389fbccbec3c9d4ee034602f19f0bc0d05ee65"}
Nov 03 18:15:18.999636 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:18.998738    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6" event={"ID":"89bf9e11-1400-4a4a-b474-f23baf5d0da5","Type":"ContainerStarted","Data":"3218dcdb99b491de999a1748c5ad6e6b1e8c087c8b8a370d1d59f82ed7b151eb"}
Nov 03 18:15:19.001010 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:19.000982    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-jgzv4" event={"ID":"e31a2c86-cff2-49ed-8f40-4020dfef4808","Type":"ContainerStarted","Data":"a9bcd93f50abfb9fc435addc415e4a2451a91dbb3dea3595d7057c9594594876"}
Nov 03 18:15:19.002815 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:19.002793    5315 generic.go:356] "Generic (PLEG): container finished" podID="5d04a4ae-44e9-4f14-ada5-855c1efe7539" containerID="a3b565e94bf5d4ff193d7a489145bf6edb645d6caabe8eefcfca68c307609b35" exitCode=60
Nov 03 18:15:19.002936 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:19.002867    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895495-r68gb" event={"ID":"5d04a4ae-44e9-4f14-ada5-855c1efe7539","Type":"ContainerDied","Data":"a3b565e94bf5d4ff193d7a489145bf6edb645d6caabe8eefcfca68c307609b35"}
Nov 03 18:15:19.003102 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:19.003083    5315 scope.go:122] "RemoveContainer" containerID="6732a4b4ec6cd747d6a75f3808ff1431e120d5c9f95c0cebf37f4b0903d8e1c9"
Nov 03 18:15:19.003369 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:19.003348    5315 scope.go:122] "RemoveContainer" containerID="a3b565e94bf5d4ff193d7a489145bf6edb645d6caabe8eefcfca68c307609b35"
Nov 03 18:15:19.003639 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:15:19.003594    5315 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-a86228-29895495-r68gb_odh-ai-gateway-infra(5d04a4ae-44e9-4f14-ada5-855c1efe7539)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895495-r68gb" podUID="5d04a4ae-44e9-4f14-ada5-855c1efe7539"
Nov 03 18:15:19.007035 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:19.006981    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895495-shccb" event={"ID":"54bd478e-0685-4ec9-8aa0-5c3bd73c56c2","Type":"ContainerDied","Data":"0109f14428fdbd14b59ecb232e0db778204832840eb291ef6732e901498b9724"}
Nov 03 18:15:19.007236 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:19.007219    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895495-shccb"
Nov 03 18:15:19.017522 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:19.017371    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb" event={"ID":"14b0b7de-c4ed-4997-ba30-72f4d501786b","Type":"ContainerStarted","Data":"a59b1240a3cb08951e1a557a8931409d4f8cd084efa072830ce65869abaef681"}
Nov 03 18:15:19.017522 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:19.017493    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb" event={"ID":"14b0b7de-c4ed-4997-ba30-72f4d501786b","Type":"ContainerStarted","Data":"547d27cdc78aa1c3e4d9d20e9575645198b3d316f93e45599dd37b59a4218e7b"}
Nov 03 18:15:19.096204 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:19.091994    5315 scope.go:122] "RemoveContainer" containerID="00733fd18a5fc690f3e550803e2616bf392d0e10347a1409de399a80a52c55e0"
Nov 03 18:15:19.114895 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:19.113488    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3a2676b-0fd6-4c39-9ed7-d468706aa9e9" path="/var/lib/kubelet/pods/e3a2676b-0fd6-4c39-9ed7-d468706aa9e9/volumes"
Nov 03 18:15:19.207989 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:19.207868    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6" podStartSLOduration=2.207807738 podStartE2EDuration="2.207807738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:15:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:15:19.19201623 +0000 UTC m=+1758.208214315" watchObservedRunningTime="2026-11-03 18:15:19.207807738 +0000 UTC m=+1758.224005808"
Nov 03 18:15:19.210147 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:19.209661    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895495-r68gb"]
Nov 03 18:15:19.421939 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:19.421912    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7a7c76-29895495-lhj7q"
Nov 03 18:15:19.487693 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:19.487646    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb" podStartSLOduration=2.487631767 podStartE2EDuration="2.487631767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:15:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:15:19.453385441 +0000 UTC m=+1758.469583512" watchObservedRunningTime="2026-11-03 18:15:19.487631767 +0000 UTC m=+1758.503829908"
Nov 03 18:15:19.502639 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:19.502583    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-80aca1-29895495-qvtbt"
Nov 03 18:15:19.606586 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:19.606495    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26d8c8d7-89c0-47af-b90c-8b0677f71886-kube-api-access-kpw8j\" (UniqueName: \"kubernetes.io/projected/26d8c8d7-89c0-47af-b90c-8b0677f71886-kube-api-access-kpw8j\") pod \"26d8c8d7-89c0-47af-b90c-8b0677f71886\" (UID: \"26d8c8d7-89c0-47af-b90c-8b0677f71886\") "
Nov 03 18:15:19.606586 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:19.606535    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d8504d9-e3e9-4542-8e76-3dbd404ed89f-kube-api-access-gj8pw\" (UniqueName: \"kubernetes.io/projected/8d8504d9-e3e9-4542-8e76-3dbd404ed89f-kube-api-access-gj8pw\") pod \"8d8504d9-e3e9-4542-8e76-3dbd404ed89f\" (UID: \"8d8504d9-e3e9-4542-8e76-3dbd404ed89f\") "
Nov 03 18:15:19.606813 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:19.606719    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26d8c8d7-89c0-47af-b90c-8b0677f71886-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26d8c8d7-89c0-47af-b90c-8b0677f71886-maas-api-service-ca\") pod \"26d8c8d7-89c0-47af-b90c-8b0677f71886\" (UID: \"26d8c8d7-89c0-47af-b90c-8b0677f71886\") "
Nov 03 18:15:19.606813 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:19.606782    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d8504d9-e3e9-4542-8e76-3dbd404ed89f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d8504d9-e3e9-4542-8e76-3dbd404ed89f-maas-api-service-ca\") pod \"8d8504d9-e3e9-4542-8e76-3dbd404ed89f\" (UID: \"8d8504d9-e3e9-4542-8e76-3dbd404ed89f\") "
Nov 03 18:15:19.607098 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:19.607067    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26d8c8d7-89c0-47af-b90c-8b0677f71886-maas-api-service-ca" pod "26d8c8d7-89c0-47af-b90c-8b0677f71886" (UID: "26d8c8d7-89c0-47af-b90c-8b0677f71886"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:19.607175 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:19.607116    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d8504d9-e3e9-4542-8e76-3dbd404ed89f-maas-api-service-ca" pod "8d8504d9-e3e9-4542-8e76-3dbd404ed89f" (UID: "8d8504d9-e3e9-4542-8e76-3dbd404ed89f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:19.607389 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:19.607366    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26d8c8d7-89c0-47af-b90c-8b0677f71886-maas-api-service-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:19.607389 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:19.607386    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d8504d9-e3e9-4542-8e76-3dbd404ed89f-maas-api-service-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:19.608896 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:19.608866    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d8504d9-e3e9-4542-8e76-3dbd404ed89f-kube-api-access-gj8pw" pod "8d8504d9-e3e9-4542-8e76-3dbd404ed89f" (UID: "8d8504d9-e3e9-4542-8e76-3dbd404ed89f"). InnerVolumeSpecName "kube-api-access-gj8pw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:19.608896 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:19.608880    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26d8c8d7-89c0-47af-b90c-8b0677f71886-kube-api-access-kpw8j" pod "26d8c8d7-89c0-47af-b90c-8b0677f71886" (UID: "26d8c8d7-89c0-47af-b90c-8b0677f71886"). InnerVolumeSpecName "kube-api-access-kpw8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:19.680781 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:19.680714    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-a86228-f656db785-jcvfr" podUID="6434783b-1272-4ccd-b5c4-5a7aeeab58c7" containerName="payload-processing" containerID="cri-o://4f62f10b9e6389ae06381c234bdf4fa35ab045d5a47f6f0aef6e0e65fce8599e" gracePeriod=25
Nov 03 18:15:19.709315 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:19.709281    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpw8j\" (UniqueName: \"kubernetes.io/projected/26d8c8d7-89c0-47af-b90c-8b0677f71886-kube-api-access-kpw8j\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:19.709315 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:19.709311    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj8pw\" (UniqueName: \"kubernetes.io/projected/8d8504d9-e3e9-4542-8e76-3dbd404ed89f-kube-api-access-gj8pw\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:19.858289 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:19.855865    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6"
Nov 03 18:15:19.858289 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:19.856000    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Nov 03 18:15:19.858289 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:19.856046    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6" podUID="89bf9e11-1400-4a4a-b474-f23baf5d0da5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Nov 03 18:15:19.877089 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:19.877051    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb"
Nov 03 18:15:19.877476 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:19.877453    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body=
Nov 03 18:15:19.877693 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:19.877669    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb" podUID="14b0b7de-c4ed-4997-ba30-72f4d501786b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused"
Nov 03 18:15:19.907305 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:19.907260    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895495-shccb"]
Nov 03 18:15:19.910520 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:19.910496    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895495-shccb"]
Nov 03 18:15:20.098709 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:20.098673    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-80aca1-29895495-qvtbt"
Nov 03 18:15:20.099138 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:20.098783    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-80aca1-29895495-qvtbt" event={"ID":"26d8c8d7-89c0-47af-b90c-8b0677f71886","Type":"ContainerDied","Data":"2a82be8d5e7829525d6503f8761e03aedd501b5734e44b700d48c899b85f0532"}
Nov 03 18:15:20.099138 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:20.098852    5315 scope.go:122] "RemoveContainer" containerID="43ce9103e2b6b78beacea4a35770da72af78ea7168b3ff7e1f07545fc55836c8"
Nov 03 18:15:20.101523 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:20.101499    5315 generic.go:356] "Generic (PLEG): container finished" podID="e31a2c86-cff2-49ed-8f40-4020dfef4808" containerID="a9bcd93f50abfb9fc435addc415e4a2451a91dbb3dea3595d7057c9594594876" exitCode=60
Nov 03 18:15:20.102128 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:20.101544    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-jgzv4" event={"ID":"e31a2c86-cff2-49ed-8f40-4020dfef4808","Type":"ContainerDied","Data":"a9bcd93f50abfb9fc435addc415e4a2451a91dbb3dea3595d7057c9594594876"}
Nov 03 18:15:20.102661 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:20.102599    5315 scope.go:122] "RemoveContainer" containerID="a9bcd93f50abfb9fc435addc415e4a2451a91dbb3dea3595d7057c9594594876"
Nov 03 18:15:20.105816 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:15:20.105788    5315 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895495-jgzv4_odh-ai-gateway-infra(e31a2c86-cff2-49ed-8f40-4020dfef4808)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-jgzv4" podUID="e31a2c86-cff2-49ed-8f40-4020dfef4808"
Nov 03 18:15:20.107570 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:20.107542    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7a7c76-29895495-lhj7q" event={"ID":"8d8504d9-e3e9-4542-8e76-3dbd404ed89f","Type":"ContainerDied","Data":"940909dfd9d4e115fe39124c3a5343c01d31b75c0a1ddf32981ddbe3ecefdcd2"}
Nov 03 18:15:20.107686 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:20.107549    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7a7c76-29895495-lhj7q"
Nov 03 18:15:20.114335 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:20.114317    5315 scope.go:122] "RemoveContainer" containerID="641a9509da3ff55fadbd9e96744036800cd32b606e9ad3de2ee1782a9abb0e8f"
Nov 03 18:15:20.204522 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:20.204391    5315 scope.go:122] "RemoveContainer" containerID="0b6f31d62b15fca0f3b0deafacbec9128fcc85da9917c8391fe435bbe819ca16"
Nov 03 18:15:20.485714 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:20.483330    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7a7c76-29895495-lhj7q"]
Nov 03 18:15:20.488893 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:20.488861    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7a7c76-29895495-lhj7q"]
Nov 03 18:15:20.518643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:20.518602    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895495-r68gb"
Nov 03 18:15:20.577002 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:20.576955    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895495-jgzv4"]
Nov 03 18:15:20.583914 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:20.583877    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d04a4ae-44e9-4f14-ada5-855c1efe7539-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d04a4ae-44e9-4f14-ada5-855c1efe7539-maas-api-service-ca\") pod \"5d04a4ae-44e9-4f14-ada5-855c1efe7539\" (UID: \"5d04a4ae-44e9-4f14-ada5-855c1efe7539\") "
Nov 03 18:15:20.584095 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:20.584020    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d04a4ae-44e9-4f14-ada5-855c1efe7539-kube-api-access-v9r5l\" (UniqueName: \"kubernetes.io/projected/5d04a4ae-44e9-4f14-ada5-855c1efe7539-kube-api-access-v9r5l\") pod \"5d04a4ae-44e9-4f14-ada5-855c1efe7539\" (UID: \"5d04a4ae-44e9-4f14-ada5-855c1efe7539\") "
Nov 03 18:15:20.584438 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:20.584394    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d04a4ae-44e9-4f14-ada5-855c1efe7539-maas-api-service-ca" pod "5d04a4ae-44e9-4f14-ada5-855c1efe7539" (UID: "5d04a4ae-44e9-4f14-ada5-855c1efe7539"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:20.586373 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:20.586348    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d04a4ae-44e9-4f14-ada5-855c1efe7539-kube-api-access-v9r5l" pod "5d04a4ae-44e9-4f14-ada5-855c1efe7539" (UID: "5d04a4ae-44e9-4f14-ada5-855c1efe7539"). InnerVolumeSpecName "kube-api-access-v9r5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:20.686251 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:20.686215    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9r5l\" (UniqueName: \"kubernetes.io/projected/5d04a4ae-44e9-4f14-ada5-855c1efe7539-kube-api-access-v9r5l\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:20.686251 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:20.686247    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d04a4ae-44e9-4f14-ada5-855c1efe7539-maas-api-service-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:20.711898 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:20.711864    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-80aca1-29895495-qvtbt"]
Nov 03 18:15:20.717723 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:20.717695    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-80aca1-29895495-qvtbt"]
Nov 03 18:15:20.862653 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:20.860522    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Nov 03 18:15:20.862653 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:20.860583    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6" podUID="89bf9e11-1400-4a4a-b474-f23baf5d0da5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Nov 03 18:15:20.862653 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:20.862546    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body=
Nov 03 18:15:20.863213 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:20.862593    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb" podUID="14b0b7de-c4ed-4997-ba30-72f4d501786b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused"
Nov 03 18:15:21.106511 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:21.106476    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26d8c8d7-89c0-47af-b90c-8b0677f71886" path="/var/lib/kubelet/pods/26d8c8d7-89c0-47af-b90c-8b0677f71886/volumes"
Nov 03 18:15:21.106935 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:21.106919    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54bd478e-0685-4ec9-8aa0-5c3bd73c56c2" path="/var/lib/kubelet/pods/54bd478e-0685-4ec9-8aa0-5c3bd73c56c2/volumes"
Nov 03 18:15:21.107298 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:21.107286    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d8504d9-e3e9-4542-8e76-3dbd404ed89f" path="/var/lib/kubelet/pods/8d8504d9-e3e9-4542-8e76-3dbd404ed89f/volumes"
Nov 03 18:15:21.180755 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:21.180728    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895495-r68gb"
Nov 03 18:15:21.181061 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:21.180725    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895495-r68gb" event={"ID":"5d04a4ae-44e9-4f14-ada5-855c1efe7539","Type":"ContainerDied","Data":"4c5e8935819a297c6c0ec12562cb7099d66e968e0e5b0e5c8edc4b30042638eb"}
Nov 03 18:15:21.181178 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:21.181109    5315 scope.go:122] "RemoveContainer" containerID="a3b565e94bf5d4ff193d7a489145bf6edb645d6caabe8eefcfca68c307609b35"
Nov 03 18:15:21.400203 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:21.400119    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895495-r68gb"]
Nov 03 18:15:21.405035 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:21.405004    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a86228-29895495-r68gb"]
Nov 03 18:15:21.496572 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:21.496520    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c" podUID="8ac59539-8ac1-466e-8d1b-b2be4c42c708" containerName="main" containerID="cri-o://b8842e0c1bf5eac4427d18af275629e52ff1535fea3e0723abc822cd02f10d77" gracePeriod=45
Nov 03 18:15:21.503233 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:21.503052    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr" podUID="df8e0b71-e876-4cea-a7ae-3532c887c660" containerName="main" containerID="cri-o://2a87abe4c0f9b40f7d2fbd81a89e003534fe58ba68921c8745660c7c03a3d6ef" gracePeriod=45
Nov 03 18:15:21.689588 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:21.689504    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-jgzv4"
Nov 03 18:15:21.705429 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:21.705384    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e31a2c86-cff2-49ed-8f40-4020dfef4808-maas-api-service-ca" pod "e31a2c86-cff2-49ed-8f40-4020dfef4808" (UID: "e31a2c86-cff2-49ed-8f40-4020dfef4808"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:21.705687 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:21.705664    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e31a2c86-cff2-49ed-8f40-4020dfef4808-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e31a2c86-cff2-49ed-8f40-4020dfef4808-maas-api-service-ca\") pod \"e31a2c86-cff2-49ed-8f40-4020dfef4808\" (UID: \"e31a2c86-cff2-49ed-8f40-4020dfef4808\") "
Nov 03 18:15:21.705804 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:21.705790    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e31a2c86-cff2-49ed-8f40-4020dfef4808-kube-api-access-5s7tp\" (UniqueName: \"kubernetes.io/projected/e31a2c86-cff2-49ed-8f40-4020dfef4808-kube-api-access-5s7tp\") pod \"e31a2c86-cff2-49ed-8f40-4020dfef4808\" (UID: \"e31a2c86-cff2-49ed-8f40-4020dfef4808\") "
Nov 03 18:15:21.706285 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:21.706224    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e31a2c86-cff2-49ed-8f40-4020dfef4808-maas-api-service-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:21.711720 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:21.711076    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e31a2c86-cff2-49ed-8f40-4020dfef4808-kube-api-access-5s7tp" pod "e31a2c86-cff2-49ed-8f40-4020dfef4808" (UID: "e31a2c86-cff2-49ed-8f40-4020dfef4808"). InnerVolumeSpecName "kube-api-access-5s7tp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:21.715636 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:21.714527    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr" podUID="df8e0b71-e876-4cea-a7ae-3532c887c660" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": dial tcp 10.133.0.46:8000: connect: connection refused"
Nov 03 18:15:21.825234 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:21.825137    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s7tp\" (UniqueName: \"kubernetes.io/projected/e31a2c86-cff2-49ed-8f40-4020dfef4808-kube-api-access-5s7tp\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:21.896037 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:21.886950    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Nov 03 18:15:21.896037 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:21.887034    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6" podUID="89bf9e11-1400-4a4a-b474-f23baf5d0da5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Nov 03 18:15:21.896037 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:21.888421    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body=
Nov 03 18:15:21.896037 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:21.888460    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb" podUID="14b0b7de-c4ed-4997-ba30-72f4d501786b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused"
Nov 03 18:15:21.993259 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:21.993168    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1720c00c-7f49c64bcf-pmktn"]
Nov 03 18:15:22.116317 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.115958    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr"
Nov 03 18:15:22.117771 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.117665    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb"]
Nov 03 18:15:22.121768 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.121696    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c"
Nov 03 18:15:22.123696 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.122873    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg"]
Nov 03 18:15:22.123696 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.123085    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg" podUID="daf3798f-da45-4287-94f6-eb32897e807b" containerName="istio-proxy" containerID="cri-o://c34d00783b27c609f5b93682395d9b1b6c44e516167c7b85d02064b2cef11dd8" gracePeriod=30
Nov 03 18:15:22.193719 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.193412    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w"]
Nov 03 18:15:22.194286 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.194088    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w" podUID="3d07e31d-20b5-47a1-901c-915565686286" containerName="istio-proxy" containerID="cri-o://1b3c165beae7fd0146d0d02033bc972b5d841ca2526f2d2bb77ff800f0adb8c9" gracePeriod=30
Nov 03 18:15:22.199089 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.199065    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6"]
Nov 03 18:15:22.201944 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.201916    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-jgzv4" event={"ID":"e31a2c86-cff2-49ed-8f40-4020dfef4808","Type":"ContainerDied","Data":"428006d9de17c40e2114647c1f3d3707dd56b62c47fe4cb0828689db0665e1a4"}
Nov 03 18:15:22.202079 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.202028    5315 scope.go:122] "RemoveContainer" containerID="a9bcd93f50abfb9fc435addc415e4a2451a91dbb3dea3595d7057c9594594876"
Nov 03 18:15:22.202929 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.202623    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-jgzv4"
Nov 03 18:15:22.205808 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.205782    5315 generic.go:356] "Generic (PLEG): container finished" podID="8ac59539-8ac1-466e-8d1b-b2be4c42c708" containerID="b8842e0c1bf5eac4427d18af275629e52ff1535fea3e0723abc822cd02f10d77" exitCode=0
Nov 03 18:15:22.205916 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.205833    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c"
Nov 03 18:15:22.205976 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.205951    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c" event={"ID":"8ac59539-8ac1-466e-8d1b-b2be4c42c708","Type":"ContainerDied","Data":"b8842e0c1bf5eac4427d18af275629e52ff1535fea3e0723abc822cd02f10d77"}
Nov 03 18:15:22.206047 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.205983    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c" event={"ID":"8ac59539-8ac1-466e-8d1b-b2be4c42c708","Type":"ContainerDied","Data":"cdb341ab0a193afbf77aaafd309c6658b1495a0a14cfbdf70ed15a15fe54dd9c"}
Nov 03 18:15:22.206759 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.206731    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1720c00c-549d8c448c-v5z86"]
Nov 03 18:15:22.207537 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.207513    5315 generic.go:356] "Generic (PLEG): container finished" podID="df8e0b71-e876-4cea-a7ae-3532c887c660" containerID="2a87abe4c0f9b40f7d2fbd81a89e003534fe58ba68921c8745660c7c03a3d6ef" exitCode=0
Nov 03 18:15:22.207789 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.207768    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb" podUID="14b0b7de-c4ed-4997-ba30-72f4d501786b" containerName="istio-proxy" containerID="cri-o://a59b1240a3cb08951e1a557a8931409d4f8cd084efa072830ce65869abaef681" gracePeriod=30
Nov 03 18:15:22.207891 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.207870    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr" event={"ID":"df8e0b71-e876-4cea-a7ae-3532c887c660","Type":"ContainerDied","Data":"2a87abe4c0f9b40f7d2fbd81a89e003534fe58ba68921c8745660c7c03a3d6ef"}
Nov 03 18:15:22.207980 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.207905    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr" event={"ID":"df8e0b71-e876-4cea-a7ae-3532c887c660","Type":"ContainerDied","Data":"4d1322b01636dfab68fe2fc5c01456363dac6c4efbd1d012d6a8bc6b0f5ae3be"}
Nov 03 18:15:22.207980 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.207959    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr"
Nov 03 18:15:22.208106 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.207956    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6" podUID="89bf9e11-1400-4a4a-b474-f23baf5d0da5" containerName="istio-proxy" containerID="cri-o://532430e4867c14237c848ad891389fbccbec3c9d4ee034602f19f0bc0d05ee65" gracePeriod=30
Nov 03 18:15:22.219957 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.219936    5315 scope.go:122] "RemoveContainer" containerID="b8842e0c1bf5eac4427d18af275629e52ff1535fea3e0723abc822cd02f10d77"
Nov 03 18:15:22.409287 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.393568    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df8e0b71-e876-4cea-a7ae-3532c887c660-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df8e0b71-e876-4cea-a7ae-3532c887c660-tmp-dir\") pod \"df8e0b71-e876-4cea-a7ae-3532c887c660\" (UID: \"df8e0b71-e876-4cea-a7ae-3532c887c660\") "
Nov 03 18:15:22.409287 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.393716    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ac59539-8ac1-466e-8d1b-b2be4c42c708-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ac59539-8ac1-466e-8d1b-b2be4c42c708-tmp-dir\") pod \"8ac59539-8ac1-466e-8d1b-b2be4c42c708\" (UID: \"8ac59539-8ac1-466e-8d1b-b2be4c42c708\") "
Nov 03 18:15:22.409287 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.393862    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ac59539-8ac1-466e-8d1b-b2be4c42c708-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ac59539-8ac1-466e-8d1b-b2be4c42c708-model-cache\") pod \"8ac59539-8ac1-466e-8d1b-b2be4c42c708\" (UID: \"8ac59539-8ac1-466e-8d1b-b2be4c42c708\") "
Nov 03 18:15:22.409287 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.393900    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ac59539-8ac1-466e-8d1b-b2be4c42c708-tls-certs\" (UniqueName: \"kubernetes.io/secret/8ac59539-8ac1-466e-8d1b-b2be4c42c708-tls-certs\") pod \"8ac59539-8ac1-466e-8d1b-b2be4c42c708\" (UID: \"8ac59539-8ac1-466e-8d1b-b2be4c42c708\") "
Nov 03 18:15:22.409287 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.394292    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ac59539-8ac1-466e-8d1b-b2be4c42c708-home\" (UniqueName: \"kubernetes.io/empty-dir/8ac59539-8ac1-466e-8d1b-b2be4c42c708-home\") pod \"8ac59539-8ac1-466e-8d1b-b2be4c42c708\" (UID: \"8ac59539-8ac1-466e-8d1b-b2be4c42c708\") "
Nov 03 18:15:22.409287 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.394349    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df8e0b71-e876-4cea-a7ae-3532c887c660-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df8e0b71-e876-4cea-a7ae-3532c887c660-model-cache\") pod \"df8e0b71-e876-4cea-a7ae-3532c887c660\" (UID: \"df8e0b71-e876-4cea-a7ae-3532c887c660\") "
Nov 03 18:15:22.409287 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.394415    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df8e0b71-e876-4cea-a7ae-3532c887c660-tls-certs\" (UniqueName: \"kubernetes.io/secret/df8e0b71-e876-4cea-a7ae-3532c887c660-tls-certs\") pod \"df8e0b71-e876-4cea-a7ae-3532c887c660\" (UID: \"df8e0b71-e876-4cea-a7ae-3532c887c660\") "
Nov 03 18:15:22.409287 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.394543    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df8e0b71-e876-4cea-a7ae-3532c887c660-kube-api-access-xdstr\" (UniqueName: \"kubernetes.io/projected/df8e0b71-e876-4cea-a7ae-3532c887c660-kube-api-access-xdstr\") pod \"df8e0b71-e876-4cea-a7ae-3532c887c660\" (UID: \"df8e0b71-e876-4cea-a7ae-3532c887c660\") "
Nov 03 18:15:22.409287 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.394885    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df8e0b71-e876-4cea-a7ae-3532c887c660-home\" (UniqueName: \"kubernetes.io/empty-dir/df8e0b71-e876-4cea-a7ae-3532c887c660-home\") pod \"df8e0b71-e876-4cea-a7ae-3532c887c660\" (UID: \"df8e0b71-e876-4cea-a7ae-3532c887c660\") "
Nov 03 18:15:22.409287 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.394932    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df8e0b71-e876-4cea-a7ae-3532c887c660-dshm\" (UniqueName: \"kubernetes.io/empty-dir/df8e0b71-e876-4cea-a7ae-3532c887c660-dshm\") pod \"df8e0b71-e876-4cea-a7ae-3532c887c660\" (UID: \"df8e0b71-e876-4cea-a7ae-3532c887c660\") "
Nov 03 18:15:22.409287 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.394981    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ac59539-8ac1-466e-8d1b-b2be4c42c708-kube-api-access-hbh7r\" (UniqueName: \"kubernetes.io/projected/8ac59539-8ac1-466e-8d1b-b2be4c42c708-kube-api-access-hbh7r\") pod \"8ac59539-8ac1-466e-8d1b-b2be4c42c708\" (UID: \"8ac59539-8ac1-466e-8d1b-b2be4c42c708\") "
Nov 03 18:15:22.409287 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.395014    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ac59539-8ac1-466e-8d1b-b2be4c42c708-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ac59539-8ac1-466e-8d1b-b2be4c42c708-dshm\") pod \"8ac59539-8ac1-466e-8d1b-b2be4c42c708\" (UID: \"8ac59539-8ac1-466e-8d1b-b2be4c42c708\") "
Nov 03 18:15:22.409287 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.408389    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ac59539-8ac1-466e-8d1b-b2be4c42c708-home" pod "8ac59539-8ac1-466e-8d1b-b2be4c42c708" (UID: "8ac59539-8ac1-466e-8d1b-b2be4c42c708"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:22.409287 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.408963    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df8e0b71-e876-4cea-a7ae-3532c887c660-tmp-dir" pod "df8e0b71-e876-4cea-a7ae-3532c887c660" (UID: "df8e0b71-e876-4cea-a7ae-3532c887c660"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:22.409287 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.409144    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ac59539-8ac1-466e-8d1b-b2be4c42c708-tmp-dir" pod "8ac59539-8ac1-466e-8d1b-b2be4c42c708" (UID: "8ac59539-8ac1-466e-8d1b-b2be4c42c708"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:22.417200 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.410246    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ac59539-8ac1-466e-8d1b-b2be4c42c708-model-cache" pod "8ac59539-8ac1-466e-8d1b-b2be4c42c708" (UID: "8ac59539-8ac1-466e-8d1b-b2be4c42c708"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:22.417200 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.411030    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df8e0b71-e876-4cea-a7ae-3532c887c660-model-cache" pod "df8e0b71-e876-4cea-a7ae-3532c887c660" (UID: "df8e0b71-e876-4cea-a7ae-3532c887c660"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:22.444508 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.431929    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df8e0b71-e876-4cea-a7ae-3532c887c660-home" pod "df8e0b71-e876-4cea-a7ae-3532c887c660" (UID: "df8e0b71-e876-4cea-a7ae-3532c887c660"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:22.561949 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.521039    5315 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df8e0b71-e876-4cea-a7ae-3532c887c660-home\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:22.561949 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.521082    5315 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df8e0b71-e876-4cea-a7ae-3532c887c660-tmp-dir\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:22.561949 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.521096    5315 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ac59539-8ac1-466e-8d1b-b2be4c42c708-tmp-dir\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:22.561949 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.521112    5315 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ac59539-8ac1-466e-8d1b-b2be4c42c708-model-cache\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:22.561949 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.521127    5315 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ac59539-8ac1-466e-8d1b-b2be4c42c708-home\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:22.561949 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.521141    5315 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df8e0b71-e876-4cea-a7ae-3532c887c660-model-cache\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:22.652687 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.625352    5315 scope.go:122] "RemoveContainer" containerID="b8842e0c1bf5eac4427d18af275629e52ff1535fea3e0723abc822cd02f10d77"
Nov 03 18:15:22.653672 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:15:22.653635    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8842e0c1bf5eac4427d18af275629e52ff1535fea3e0723abc822cd02f10d77\": container with ID starting with b8842e0c1bf5eac4427d18af275629e52ff1535fea3e0723abc822cd02f10d77 not found: ID does not exist" containerID="b8842e0c1bf5eac4427d18af275629e52ff1535fea3e0723abc822cd02f10d77"
Nov 03 18:15:22.653979 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.653950    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8842e0c1bf5eac4427d18af275629e52ff1535fea3e0723abc822cd02f10d77"} err="failed to get container status \"b8842e0c1bf5eac4427d18af275629e52ff1535fea3e0723abc822cd02f10d77\": rpc error: code = NotFound desc = could not find container \"b8842e0c1bf5eac4427d18af275629e52ff1535fea3e0723abc822cd02f10d77\": container with ID starting with b8842e0c1bf5eac4427d18af275629e52ff1535fea3e0723abc822cd02f10d77 not found: ID does not exist"
Nov 03 18:15:22.654122 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.654108    5315 scope.go:122] "RemoveContainer" containerID="2a87abe4c0f9b40f7d2fbd81a89e003534fe58ba68921c8745660c7c03a3d6ef"
Nov 03 18:15:22.693764 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.689022    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ac59539-8ac1-466e-8d1b-b2be4c42c708-kube-api-access-hbh7r" pod "8ac59539-8ac1-466e-8d1b-b2be4c42c708" (UID: "8ac59539-8ac1-466e-8d1b-b2be4c42c708"). InnerVolumeSpecName "kube-api-access-hbh7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:22.735632 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.735544    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbh7r\" (UniqueName: \"kubernetes.io/projected/8ac59539-8ac1-466e-8d1b-b2be4c42c708-kube-api-access-hbh7r\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:22.785893 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.785818    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df8e0b71-e876-4cea-a7ae-3532c887c660-dshm" pod "df8e0b71-e876-4cea-a7ae-3532c887c660" (UID: "df8e0b71-e876-4cea-a7ae-3532c887c660"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:22.788929 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.788861    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ac59539-8ac1-466e-8d1b-b2be4c42c708-tls-certs" pod "8ac59539-8ac1-466e-8d1b-b2be4c42c708" (UID: "8ac59539-8ac1-466e-8d1b-b2be4c42c708"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:15:22.798678 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.798574    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ac59539-8ac1-466e-8d1b-b2be4c42c708-dshm" pod "8ac59539-8ac1-466e-8d1b-b2be4c42c708" (UID: "8ac59539-8ac1-466e-8d1b-b2be4c42c708"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:22.798818 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.798698    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df8e0b71-e876-4cea-a7ae-3532c887c660-kube-api-access-xdstr" pod "df8e0b71-e876-4cea-a7ae-3532c887c660" (UID: "df8e0b71-e876-4cea-a7ae-3532c887c660"). InnerVolumeSpecName "kube-api-access-xdstr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:22.801514 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.801488    5315 scope.go:122] "RemoveContainer" containerID="2a87abe4c0f9b40f7d2fbd81a89e003534fe58ba68921c8745660c7c03a3d6ef"
Nov 03 18:15:22.806019 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:15:22.805986    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a87abe4c0f9b40f7d2fbd81a89e003534fe58ba68921c8745660c7c03a3d6ef\": container with ID starting with 2a87abe4c0f9b40f7d2fbd81a89e003534fe58ba68921c8745660c7c03a3d6ef not found: ID does not exist" containerID="2a87abe4c0f9b40f7d2fbd81a89e003534fe58ba68921c8745660c7c03a3d6ef"
Nov 03 18:15:22.806157 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.806132    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a87abe4c0f9b40f7d2fbd81a89e003534fe58ba68921c8745660c7c03a3d6ef"} err="failed to get container status \"2a87abe4c0f9b40f7d2fbd81a89e003534fe58ba68921c8745660c7c03a3d6ef\": rpc error: code = NotFound desc = could not find container \"2a87abe4c0f9b40f7d2fbd81a89e003534fe58ba68921c8745660c7c03a3d6ef\": container with ID starting with 2a87abe4c0f9b40f7d2fbd81a89e003534fe58ba68921c8745660c7c03a3d6ef not found: ID does not exist"
Nov 03 18:15:22.807229 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.807201    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df8e0b71-e876-4cea-a7ae-3532c887c660-tls-certs" pod "df8e0b71-e876-4cea-a7ae-3532c887c660" (UID: "df8e0b71-e876-4cea-a7ae-3532c887c660"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:15:22.893112 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.853182    5315 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df8e0b71-e876-4cea-a7ae-3532c887c660-dshm\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:22.893112 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.853220    5315 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ac59539-8ac1-466e-8d1b-b2be4c42c708-dshm\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:22.893112 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.853235    5315 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ac59539-8ac1-466e-8d1b-b2be4c42c708-tls-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:22.893112 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.853248    5315 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df8e0b71-e876-4cea-a7ae-3532c887c660-tls-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:22.893112 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:22.853264    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdstr\" (UniqueName: \"kubernetes.io/projected/df8e0b71-e876-4cea-a7ae-3532c887c660-kube-api-access-xdstr\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:23.048805 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:23.048750    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c" podUID="8ac59539-8ac1-466e-8d1b-b2be4c42c708" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Nov 03 18:15:23.107497 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:23.107451    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d04a4ae-44e9-4f14-ada5-855c1efe7539" path="/var/lib/kubelet/pods/5d04a4ae-44e9-4f14-ada5-855c1efe7539/volumes"
Nov 03 18:15:23.379823 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:23.379668    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895495-jgzv4"]
Nov 03 18:15:23.383599 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:23.383568    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895495-jgzv4"]
Nov 03 18:15:23.650693 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:23.648665    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr"]
Nov 03 18:15:23.684694 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:23.684654    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-a86228/premium-w4-a86228-kserve-7df468f8ff-54nfr"]
Nov 03 18:15:23.765705 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:23.765663    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c"]
Nov 03 18:15:23.785668 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:23.785630    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-a86228/distinct-w4-a86228-kserve-58d9bf9c4-44l8c"]
Nov 03 18:15:24.611482 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:15:24.611447    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:15:24.616351 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:24.616320    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-a86228-f656db785-jcvfr" podUID="6434783b-1272-4ccd-b5c4-5a7aeeab58c7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:15:25.104390 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:25.104357    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ac59539-8ac1-466e-8d1b-b2be4c42c708" path="/var/lib/kubelet/pods/8ac59539-8ac1-466e-8d1b-b2be4c42c708/volumes"
Nov 03 18:15:25.104880 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:25.104864    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df8e0b71-e876-4cea-a7ae-3532c887c660" path="/var/lib/kubelet/pods/df8e0b71-e876-4cea-a7ae-3532c887c660/volumes"
Nov 03 18:15:25.105233 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:25.105220    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e31a2c86-cff2-49ed-8f40-4020dfef4808" path="/var/lib/kubelet/pods/e31a2c86-cff2-49ed-8f40-4020dfef4808/volumes"
Nov 03 18:15:26.994059 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:26.994006    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-1720c00c-7f49c64bcf-pmktn" podUID="e2ad7926-76d5-48b1-8311-d3adb5742fd2" containerName="payload-processing" containerID="cri-o://0aad1bbb8c55382336f0f49a62296ccd8b182596d6bd82407766e0ba6afe901f" gracePeriod=25
Nov 03 18:15:27.207375 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.207335    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-1720c00c-549d8c448c-v5z86" podUID="c06ba561-05ef-49c8-b410-6aa8cb926b93" containerName="payload-pre-processing" containerID="cri-o://898f7e8995a9ba67aeaa853d027be769612a9ecbabf900a8efbf59446f782763" gracePeriod=25
Nov 03 18:15:27.318919 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.318877    5315 generic.go:356] "Generic (PLEG): container finished" podID="daf3798f-da45-4287-94f6-eb32897e807b" containerID="c34d00783b27c609f5b93682395d9b1b6c44e516167c7b85d02064b2cef11dd8" exitCode=0
Nov 03 18:15:27.319182 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.319159    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg" event={"ID":"daf3798f-da45-4287-94f6-eb32897e807b","Type":"ContainerDied","Data":"c34d00783b27c609f5b93682395d9b1b6c44e516167c7b85d02064b2cef11dd8"}
Nov 03 18:15:27.467124 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.466315    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg"
Nov 03 18:15:27.562695 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.562655    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daf3798f-da45-4287-94f6-eb32897e807b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daf3798f-da45-4287-94f6-eb32897e807b-istiod-ca-cert\") pod \"daf3798f-da45-4287-94f6-eb32897e807b\" (UID: \"daf3798f-da45-4287-94f6-eb32897e807b\") "
Nov 03 18:15:27.562872 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.562731    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daf3798f-da45-4287-94f6-eb32897e807b-istio-token\" (UniqueName: \"kubernetes.io/projected/daf3798f-da45-4287-94f6-eb32897e807b-istio-token\") pod \"daf3798f-da45-4287-94f6-eb32897e807b\" (UID: \"daf3798f-da45-4287-94f6-eb32897e807b\") "
Nov 03 18:15:27.562872 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.562767    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daf3798f-da45-4287-94f6-eb32897e807b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf3798f-da45-4287-94f6-eb32897e807b-workload-socket\") pod \"daf3798f-da45-4287-94f6-eb32897e807b\" (UID: \"daf3798f-da45-4287-94f6-eb32897e807b\") "
Nov 03 18:15:27.562872 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.562802    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daf3798f-da45-4287-94f6-eb32897e807b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daf3798f-da45-4287-94f6-eb32897e807b-istio-data\") pod \"daf3798f-da45-4287-94f6-eb32897e807b\" (UID: \"daf3798f-da45-4287-94f6-eb32897e807b\") "
Nov 03 18:15:27.562872 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.562834    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daf3798f-da45-4287-94f6-eb32897e807b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf3798f-da45-4287-94f6-eb32897e807b-credential-socket\") pod \"daf3798f-da45-4287-94f6-eb32897e807b\" (UID: \"daf3798f-da45-4287-94f6-eb32897e807b\") "
Nov 03 18:15:27.563030 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.562933    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/daf3798f-da45-4287-94f6-eb32897e807b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daf3798f-da45-4287-94f6-eb32897e807b-istio-podinfo\") pod \"daf3798f-da45-4287-94f6-eb32897e807b\" (UID: \"daf3798f-da45-4287-94f6-eb32897e807b\") "
Nov 03 18:15:27.563030 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.562992    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daf3798f-da45-4287-94f6-eb32897e807b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daf3798f-da45-4287-94f6-eb32897e807b-istio-envoy\") pod \"daf3798f-da45-4287-94f6-eb32897e807b\" (UID: \"daf3798f-da45-4287-94f6-eb32897e807b\") "
Nov 03 18:15:27.563111 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.563030    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daf3798f-da45-4287-94f6-eb32897e807b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daf3798f-da45-4287-94f6-eb32897e807b-workload-certs\") pod \"daf3798f-da45-4287-94f6-eb32897e807b\" (UID: \"daf3798f-da45-4287-94f6-eb32897e807b\") "
Nov 03 18:15:27.563111 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.563079    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daf3798f-da45-4287-94f6-eb32897e807b-kube-api-access-6hppr\" (UniqueName: \"kubernetes.io/projected/daf3798f-da45-4287-94f6-eb32897e807b-kube-api-access-6hppr\") pod \"daf3798f-da45-4287-94f6-eb32897e807b\" (UID: \"daf3798f-da45-4287-94f6-eb32897e807b\") "
Nov 03 18:15:27.563922 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.563891    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daf3798f-da45-4287-94f6-eb32897e807b-istio-data" pod "daf3798f-da45-4287-94f6-eb32897e807b" (UID: "daf3798f-da45-4287-94f6-eb32897e807b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:27.564046 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.564006    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daf3798f-da45-4287-94f6-eb32897e807b-workload-socket" pod "daf3798f-da45-4287-94f6-eb32897e807b" (UID: "daf3798f-da45-4287-94f6-eb32897e807b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:27.564116 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.564060    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daf3798f-da45-4287-94f6-eb32897e807b-credential-socket" pod "daf3798f-da45-4287-94f6-eb32897e807b" (UID: "daf3798f-da45-4287-94f6-eb32897e807b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:27.564202 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.564183    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daf3798f-da45-4287-94f6-eb32897e807b-workload-certs" pod "daf3798f-da45-4287-94f6-eb32897e807b" (UID: "daf3798f-da45-4287-94f6-eb32897e807b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:27.564416 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.564386    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daf3798f-da45-4287-94f6-eb32897e807b-istiod-ca-cert" pod "daf3798f-da45-4287-94f6-eb32897e807b" (UID: "daf3798f-da45-4287-94f6-eb32897e807b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:27.566297 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.566273    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daf3798f-da45-4287-94f6-eb32897e807b-istio-token" pod "daf3798f-da45-4287-94f6-eb32897e807b" (UID: "daf3798f-da45-4287-94f6-eb32897e807b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:27.566575 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.566543    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daf3798f-da45-4287-94f6-eb32897e807b-kube-api-access-6hppr" pod "daf3798f-da45-4287-94f6-eb32897e807b" (UID: "daf3798f-da45-4287-94f6-eb32897e807b"). InnerVolumeSpecName "kube-api-access-6hppr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:27.566846 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.566816    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daf3798f-da45-4287-94f6-eb32897e807b-istio-envoy" pod "daf3798f-da45-4287-94f6-eb32897e807b" (UID: "daf3798f-da45-4287-94f6-eb32897e807b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:27.567114 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.567097    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/daf3798f-da45-4287-94f6-eb32897e807b-istio-podinfo" pod "daf3798f-da45-4287-94f6-eb32897e807b" (UID: "daf3798f-da45-4287-94f6-eb32897e807b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:15:27.589814 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.589785    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w"
Nov 03 18:15:27.664935 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.664900    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daf3798f-da45-4287-94f6-eb32897e807b-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:27.665170 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.665154    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf3798f-da45-4287-94f6-eb32897e807b-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:27.665263 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.665251    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daf3798f-da45-4287-94f6-eb32897e807b-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:27.665343 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.665332    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf3798f-da45-4287-94f6-eb32897e807b-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:27.665432 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.665420    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daf3798f-da45-4287-94f6-eb32897e807b-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:27.665520 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.665509    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daf3798f-da45-4287-94f6-eb32897e807b-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:27.665596 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.665586    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daf3798f-da45-4287-94f6-eb32897e807b-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:27.665702 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.665688    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hppr\" (UniqueName: \"kubernetes.io/projected/daf3798f-da45-4287-94f6-eb32897e807b-kube-api-access-6hppr\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:27.665777 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.665767    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daf3798f-da45-4287-94f6-eb32897e807b-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:27.766946 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.766907    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d07e31d-20b5-47a1-901c-915565686286-istio-token\" (UniqueName: \"kubernetes.io/projected/3d07e31d-20b5-47a1-901c-915565686286-istio-token\") pod \"3d07e31d-20b5-47a1-901c-915565686286\" (UID: \"3d07e31d-20b5-47a1-901c-915565686286\") "
Nov 03 18:15:27.767154 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.766983    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d07e31d-20b5-47a1-901c-915565686286-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d07e31d-20b5-47a1-901c-915565686286-istio-envoy\") pod \"3d07e31d-20b5-47a1-901c-915565686286\" (UID: \"3d07e31d-20b5-47a1-901c-915565686286\") "
Nov 03 18:15:27.767214 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.767159    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d07e31d-20b5-47a1-901c-915565686286-kube-api-access-nrshg\" (UniqueName: \"kubernetes.io/projected/3d07e31d-20b5-47a1-901c-915565686286-kube-api-access-nrshg\") pod \"3d07e31d-20b5-47a1-901c-915565686286\" (UID: \"3d07e31d-20b5-47a1-901c-915565686286\") "
Nov 03 18:15:27.767640 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.767247    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d07e31d-20b5-47a1-901c-915565686286-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d07e31d-20b5-47a1-901c-915565686286-workload-certs\") pod \"3d07e31d-20b5-47a1-901c-915565686286\" (UID: \"3d07e31d-20b5-47a1-901c-915565686286\") "
Nov 03 18:15:27.767756 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.767553    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d07e31d-20b5-47a1-901c-915565686286-workload-certs" pod "3d07e31d-20b5-47a1-901c-915565686286" (UID: "3d07e31d-20b5-47a1-901c-915565686286"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:27.768708 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.768684    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d07e31d-20b5-47a1-901c-915565686286-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d07e31d-20b5-47a1-901c-915565686286-istiod-ca-cert\") pod \"3d07e31d-20b5-47a1-901c-915565686286\" (UID: \"3d07e31d-20b5-47a1-901c-915565686286\") "
Nov 03 18:15:27.768796 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.768769    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d07e31d-20b5-47a1-901c-915565686286-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d07e31d-20b5-47a1-901c-915565686286-istio-podinfo\") pod \"3d07e31d-20b5-47a1-901c-915565686286\" (UID: \"3d07e31d-20b5-47a1-901c-915565686286\") "
Nov 03 18:15:27.768859 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.768829    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d07e31d-20b5-47a1-901c-915565686286-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d07e31d-20b5-47a1-901c-915565686286-workload-socket\") pod \"3d07e31d-20b5-47a1-901c-915565686286\" (UID: \"3d07e31d-20b5-47a1-901c-915565686286\") "
Nov 03 18:15:27.768928 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.768874    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d07e31d-20b5-47a1-901c-915565686286-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d07e31d-20b5-47a1-901c-915565686286-istio-data\") pod \"3d07e31d-20b5-47a1-901c-915565686286\" (UID: \"3d07e31d-20b5-47a1-901c-915565686286\") "
Nov 03 18:15:27.769132 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.769109    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d07e31d-20b5-47a1-901c-915565686286-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d07e31d-20b5-47a1-901c-915565686286-credential-socket\") pod \"3d07e31d-20b5-47a1-901c-915565686286\" (UID: \"3d07e31d-20b5-47a1-901c-915565686286\") "
Nov 03 18:15:27.769208 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.769154    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d07e31d-20b5-47a1-901c-915565686286-workload-socket" pod "3d07e31d-20b5-47a1-901c-915565686286" (UID: "3d07e31d-20b5-47a1-901c-915565686286"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:27.769291 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.769250    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d07e31d-20b5-47a1-901c-915565686286-istio-data" pod "3d07e31d-20b5-47a1-901c-915565686286" (UID: "3d07e31d-20b5-47a1-901c-915565686286"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:27.769583 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.769560    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d07e31d-20b5-47a1-901c-915565686286-credential-socket" pod "3d07e31d-20b5-47a1-901c-915565686286" (UID: "3d07e31d-20b5-47a1-901c-915565686286"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:27.769930 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.769776    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d07e31d-20b5-47a1-901c-915565686286-istiod-ca-cert" pod "3d07e31d-20b5-47a1-901c-915565686286" (UID: "3d07e31d-20b5-47a1-901c-915565686286"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:27.769930 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.769792    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d07e31d-20b5-47a1-901c-915565686286-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:27.769930 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.769818    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d07e31d-20b5-47a1-901c-915565686286-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:27.769930 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.769834    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d07e31d-20b5-47a1-901c-915565686286-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:27.769930 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.769847    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d07e31d-20b5-47a1-901c-915565686286-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:27.770715 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.770691    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d07e31d-20b5-47a1-901c-915565686286-istio-envoy" pod "3d07e31d-20b5-47a1-901c-915565686286" (UID: "3d07e31d-20b5-47a1-901c-915565686286"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:27.770843 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.770767    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d07e31d-20b5-47a1-901c-915565686286-istio-token" pod "3d07e31d-20b5-47a1-901c-915565686286" (UID: "3d07e31d-20b5-47a1-901c-915565686286"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:27.770843 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.770792    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d07e31d-20b5-47a1-901c-915565686286-kube-api-access-nrshg" pod "3d07e31d-20b5-47a1-901c-915565686286" (UID: "3d07e31d-20b5-47a1-901c-915565686286"). InnerVolumeSpecName "kube-api-access-nrshg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:27.771137 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.771117    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d07e31d-20b5-47a1-901c-915565686286-istio-podinfo" pod "3d07e31d-20b5-47a1-901c-915565686286" (UID: "3d07e31d-20b5-47a1-901c-915565686286"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:15:27.871277 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.871186    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d07e31d-20b5-47a1-901c-915565686286-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:27.871277 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.871223    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d07e31d-20b5-47a1-901c-915565686286-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:27.871277 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.871235    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrshg\" (UniqueName: \"kubernetes.io/projected/3d07e31d-20b5-47a1-901c-915565686286-kube-api-access-nrshg\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:27.871277 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.871246    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d07e31d-20b5-47a1-901c-915565686286-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:27.871277 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:27.871256    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d07e31d-20b5-47a1-901c-915565686286-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:28.353927 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:28.353889    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg"
Nov 03 18:15:28.354433 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:28.353918    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg" event={"ID":"daf3798f-da45-4287-94f6-eb32897e807b","Type":"ContainerDied","Data":"ff3ccaaf90c624530731dccba5a0c8d9ff1936269c1f7d2a2a473f1b5bdcd712"}
Nov 03 18:15:28.354433 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:28.354000    5315 scope.go:122] "RemoveContainer" containerID="c34d00783b27c609f5b93682395d9b1b6c44e516167c7b85d02064b2cef11dd8"
Nov 03 18:15:28.356293 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:28.356269    5315 generic.go:356] "Generic (PLEG): container finished" podID="3d07e31d-20b5-47a1-901c-915565686286" containerID="1b3c165beae7fd0146d0d02033bc972b5d841ca2526f2d2bb77ff800f0adb8c9" exitCode=0
Nov 03 18:15:28.356661 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:28.356303    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w"
Nov 03 18:15:28.356661 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:28.356325    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w" event={"ID":"3d07e31d-20b5-47a1-901c-915565686286","Type":"ContainerDied","Data":"1b3c165beae7fd0146d0d02033bc972b5d841ca2526f2d2bb77ff800f0adb8c9"}
Nov 03 18:15:28.356813 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:28.356670    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w" event={"ID":"3d07e31d-20b5-47a1-901c-915565686286","Type":"ContainerDied","Data":"80114b976073b431c77e95a5ae67e3a6c3d764f94d6057e552b1e0928100a54b"}
Nov 03 18:15:28.367204 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:28.367185    5315 scope.go:122] "RemoveContainer" containerID="1b3c165beae7fd0146d0d02033bc972b5d841ca2526f2d2bb77ff800f0adb8c9"
Nov 03 18:15:28.377147 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:28.377124    5315 scope.go:122] "RemoveContainer" containerID="1b3c165beae7fd0146d0d02033bc972b5d841ca2526f2d2bb77ff800f0adb8c9"
Nov 03 18:15:28.377423 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:15:28.377405    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b3c165beae7fd0146d0d02033bc972b5d841ca2526f2d2bb77ff800f0adb8c9\": container with ID starting with 1b3c165beae7fd0146d0d02033bc972b5d841ca2526f2d2bb77ff800f0adb8c9 not found: ID does not exist" containerID="1b3c165beae7fd0146d0d02033bc972b5d841ca2526f2d2bb77ff800f0adb8c9"
Nov 03 18:15:28.377502 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:28.377431    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b3c165beae7fd0146d0d02033bc972b5d841ca2526f2d2bb77ff800f0adb8c9"} err="failed to get container status \"1b3c165beae7fd0146d0d02033bc972b5d841ca2526f2d2bb77ff800f0adb8c9\": rpc error: code = NotFound desc = could not find container \"1b3c165beae7fd0146d0d02033bc972b5d841ca2526f2d2bb77ff800f0adb8c9\": container with ID starting with 1b3c165beae7fd0146d0d02033bc972b5d841ca2526f2d2bb77ff800f0adb8c9 not found: ID does not exist"
Nov 03 18:15:28.508804 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:28.508690    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w"]
Nov 03 18:15:28.515001 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:28.514968    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w"]
Nov 03 18:15:28.549091 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:28.548689    5315 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:15:28.549091 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:28.548764    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-h9h7w" podUID="3d07e31d-20b5-47a1-901c-915565686286" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:15:28.619260 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:28.619215    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg"]
Nov 03 18:15:28.622734 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:28.622701    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bn7dg"]
Nov 03 18:15:29.106141 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:29.106110    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d07e31d-20b5-47a1-901c-915565686286" path="/var/lib/kubelet/pods/3d07e31d-20b5-47a1-901c-915565686286/volumes"
Nov 03 18:15:29.106685 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:29.106669    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daf3798f-da45-4287-94f6-eb32897e807b" path="/var/lib/kubelet/pods/daf3798f-da45-4287-94f6-eb32897e807b/volumes"
Nov 03 18:15:32.201496 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.201113    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb"
Nov 03 18:15:32.260631 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.259344    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6"
Nov 03 18:15:32.265013 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.264988    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89bf9e11-1400-4a4a-b474-f23baf5d0da5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89bf9e11-1400-4a4a-b474-f23baf5d0da5-credential-socket\") pod \"89bf9e11-1400-4a4a-b474-f23baf5d0da5\" (UID: \"89bf9e11-1400-4a4a-b474-f23baf5d0da5\") "
Nov 03 18:15:32.265155 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.265021    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89bf9e11-1400-4a4a-b474-f23baf5d0da5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89bf9e11-1400-4a4a-b474-f23baf5d0da5-istio-envoy\") pod \"89bf9e11-1400-4a4a-b474-f23baf5d0da5\" (UID: \"89bf9e11-1400-4a4a-b474-f23baf5d0da5\") "
Nov 03 18:15:32.265155 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.265082    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/14b0b7de-c4ed-4997-ba30-72f4d501786b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14b0b7de-c4ed-4997-ba30-72f4d501786b-istio-podinfo\") pod \"14b0b7de-c4ed-4997-ba30-72f4d501786b\" (UID: \"14b0b7de-c4ed-4997-ba30-72f4d501786b\") "
Nov 03 18:15:32.265155 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.265107    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89bf9e11-1400-4a4a-b474-f23baf5d0da5-istio-token\" (UniqueName: \"kubernetes.io/projected/89bf9e11-1400-4a4a-b474-f23baf5d0da5-istio-token\") pod \"89bf9e11-1400-4a4a-b474-f23baf5d0da5\" (UID: \"89bf9e11-1400-4a4a-b474-f23baf5d0da5\") "
Nov 03 18:15:32.265155 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.265126    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14b0b7de-c4ed-4997-ba30-72f4d501786b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b0b7de-c4ed-4997-ba30-72f4d501786b-workload-socket\") pod \"14b0b7de-c4ed-4997-ba30-72f4d501786b\" (UID: \"14b0b7de-c4ed-4997-ba30-72f4d501786b\") "
Nov 03 18:15:32.265388 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.265168    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14b0b7de-c4ed-4997-ba30-72f4d501786b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14b0b7de-c4ed-4997-ba30-72f4d501786b-istio-envoy\") pod \"14b0b7de-c4ed-4997-ba30-72f4d501786b\" (UID: \"14b0b7de-c4ed-4997-ba30-72f4d501786b\") "
Nov 03 18:15:32.265388 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.265190    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89bf9e11-1400-4a4a-b474-f23baf5d0da5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89bf9e11-1400-4a4a-b474-f23baf5d0da5-workload-socket\") pod \"89bf9e11-1400-4a4a-b474-f23baf5d0da5\" (UID: \"89bf9e11-1400-4a4a-b474-f23baf5d0da5\") "
Nov 03 18:15:32.265388 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.265217    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14b0b7de-c4ed-4997-ba30-72f4d501786b-kube-api-access-cpddv\" (UniqueName: \"kubernetes.io/projected/14b0b7de-c4ed-4997-ba30-72f4d501786b-kube-api-access-cpddv\") pod \"14b0b7de-c4ed-4997-ba30-72f4d501786b\" (UID: \"14b0b7de-c4ed-4997-ba30-72f4d501786b\") "
Nov 03 18:15:32.265388 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.265277    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89bf9e11-1400-4a4a-b474-f23baf5d0da5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89bf9e11-1400-4a4a-b474-f23baf5d0da5-workload-certs\") pod \"89bf9e11-1400-4a4a-b474-f23baf5d0da5\" (UID: \"89bf9e11-1400-4a4a-b474-f23baf5d0da5\") "
Nov 03 18:15:32.265388 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.265273    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89bf9e11-1400-4a4a-b474-f23baf5d0da5-credential-socket" pod "89bf9e11-1400-4a4a-b474-f23baf5d0da5" (UID: "89bf9e11-1400-4a4a-b474-f23baf5d0da5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:32.265388 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.265299    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89bf9e11-1400-4a4a-b474-f23baf5d0da5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89bf9e11-1400-4a4a-b474-f23baf5d0da5-istiod-ca-cert\") pod \"89bf9e11-1400-4a4a-b474-f23baf5d0da5\" (UID: \"89bf9e11-1400-4a4a-b474-f23baf5d0da5\") "
Nov 03 18:15:32.265388 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.265317    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14b0b7de-c4ed-4997-ba30-72f4d501786b-istio-token\" (UniqueName: \"kubernetes.io/projected/14b0b7de-c4ed-4997-ba30-72f4d501786b-istio-token\") pod \"14b0b7de-c4ed-4997-ba30-72f4d501786b\" (UID: \"14b0b7de-c4ed-4997-ba30-72f4d501786b\") "
Nov 03 18:15:32.265388 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.265355    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14b0b7de-c4ed-4997-ba30-72f4d501786b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14b0b7de-c4ed-4997-ba30-72f4d501786b-workload-certs\") pod \"14b0b7de-c4ed-4997-ba30-72f4d501786b\" (UID: \"14b0b7de-c4ed-4997-ba30-72f4d501786b\") "
Nov 03 18:15:32.265388 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.265377    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89bf9e11-1400-4a4a-b474-f23baf5d0da5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89bf9e11-1400-4a4a-b474-f23baf5d0da5-istio-data\") pod \"89bf9e11-1400-4a4a-b474-f23baf5d0da5\" (UID: \"89bf9e11-1400-4a4a-b474-f23baf5d0da5\") "
Nov 03 18:15:32.265976 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.265419    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89bf9e11-1400-4a4a-b474-f23baf5d0da5-kube-api-access-h5mv7\" (UniqueName: \"kubernetes.io/projected/89bf9e11-1400-4a4a-b474-f23baf5d0da5-kube-api-access-h5mv7\") pod \"89bf9e11-1400-4a4a-b474-f23baf5d0da5\" (UID: \"89bf9e11-1400-4a4a-b474-f23baf5d0da5\") "
Nov 03 18:15:32.265976 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.265452    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89bf9e11-1400-4a4a-b474-f23baf5d0da5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89bf9e11-1400-4a4a-b474-f23baf5d0da5-istio-podinfo\") pod \"89bf9e11-1400-4a4a-b474-f23baf5d0da5\" (UID: \"89bf9e11-1400-4a4a-b474-f23baf5d0da5\") "
Nov 03 18:15:32.265976 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.265481    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14b0b7de-c4ed-4997-ba30-72f4d501786b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14b0b7de-c4ed-4997-ba30-72f4d501786b-istiod-ca-cert\") pod \"14b0b7de-c4ed-4997-ba30-72f4d501786b\" (UID: \"14b0b7de-c4ed-4997-ba30-72f4d501786b\") "
Nov 03 18:15:32.265976 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.265483    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b0b7de-c4ed-4997-ba30-72f4d501786b-workload-socket" pod "14b0b7de-c4ed-4997-ba30-72f4d501786b" (UID: "14b0b7de-c4ed-4997-ba30-72f4d501786b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:32.265976 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.265519    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14b0b7de-c4ed-4997-ba30-72f4d501786b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14b0b7de-c4ed-4997-ba30-72f4d501786b-istio-data\") pod \"14b0b7de-c4ed-4997-ba30-72f4d501786b\" (UID: \"14b0b7de-c4ed-4997-ba30-72f4d501786b\") "
Nov 03 18:15:32.265976 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.265543    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14b0b7de-c4ed-4997-ba30-72f4d501786b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b0b7de-c4ed-4997-ba30-72f4d501786b-credential-socket\") pod \"14b0b7de-c4ed-4997-ba30-72f4d501786b\" (UID: \"14b0b7de-c4ed-4997-ba30-72f4d501786b\") "
Nov 03 18:15:32.266358 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.266139    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b0b7de-c4ed-4997-ba30-72f4d501786b-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:32.266358 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.266165    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89bf9e11-1400-4a4a-b474-f23baf5d0da5-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:32.266463 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.266378    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b0b7de-c4ed-4997-ba30-72f4d501786b-credential-socket" pod "14b0b7de-c4ed-4997-ba30-72f4d501786b" (UID: "14b0b7de-c4ed-4997-ba30-72f4d501786b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:32.266582 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.266553    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b0b7de-c4ed-4997-ba30-72f4d501786b-workload-certs" pod "14b0b7de-c4ed-4997-ba30-72f4d501786b" (UID: "14b0b7de-c4ed-4997-ba30-72f4d501786b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:32.266830 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.266729    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89bf9e11-1400-4a4a-b474-f23baf5d0da5-workload-certs" pod "89bf9e11-1400-4a4a-b474-f23baf5d0da5" (UID: "89bf9e11-1400-4a4a-b474-f23baf5d0da5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:32.266938 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.266831    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89bf9e11-1400-4a4a-b474-f23baf5d0da5-istio-data" pod "89bf9e11-1400-4a4a-b474-f23baf5d0da5" (UID: "89bf9e11-1400-4a4a-b474-f23baf5d0da5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:32.267058 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.267001    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b0b7de-c4ed-4997-ba30-72f4d501786b-istio-data" pod "14b0b7de-c4ed-4997-ba30-72f4d501786b" (UID: "14b0b7de-c4ed-4997-ba30-72f4d501786b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:32.267271 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.267189    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14b0b7de-c4ed-4997-ba30-72f4d501786b-istiod-ca-cert" pod "14b0b7de-c4ed-4997-ba30-72f4d501786b" (UID: "14b0b7de-c4ed-4997-ba30-72f4d501786b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:32.267271 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.267194    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89bf9e11-1400-4a4a-b474-f23baf5d0da5-workload-socket" pod "89bf9e11-1400-4a4a-b474-f23baf5d0da5" (UID: "89bf9e11-1400-4a4a-b474-f23baf5d0da5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:32.268734 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.267468    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89bf9e11-1400-4a4a-b474-f23baf5d0da5-istiod-ca-cert" pod "89bf9e11-1400-4a4a-b474-f23baf5d0da5" (UID: "89bf9e11-1400-4a4a-b474-f23baf5d0da5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:32.268862 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.268723    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14b0b7de-c4ed-4997-ba30-72f4d501786b-istio-token" pod "14b0b7de-c4ed-4997-ba30-72f4d501786b" (UID: "14b0b7de-c4ed-4997-ba30-72f4d501786b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:32.268991 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.268967    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89bf9e11-1400-4a4a-b474-f23baf5d0da5-kube-api-access-h5mv7" pod "89bf9e11-1400-4a4a-b474-f23baf5d0da5" (UID: "89bf9e11-1400-4a4a-b474-f23baf5d0da5"). InnerVolumeSpecName "kube-api-access-h5mv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:32.269301 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.269253    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b0b7de-c4ed-4997-ba30-72f4d501786b-istio-envoy" pod "14b0b7de-c4ed-4997-ba30-72f4d501786b" (UID: "14b0b7de-c4ed-4997-ba30-72f4d501786b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:32.269301 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.269290    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14b0b7de-c4ed-4997-ba30-72f4d501786b-istio-podinfo" pod "14b0b7de-c4ed-4997-ba30-72f4d501786b" (UID: "14b0b7de-c4ed-4997-ba30-72f4d501786b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:15:32.269702 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.269681    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89bf9e11-1400-4a4a-b474-f23baf5d0da5-istio-token" pod "89bf9e11-1400-4a4a-b474-f23baf5d0da5" (UID: "89bf9e11-1400-4a4a-b474-f23baf5d0da5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:32.270020 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.269989    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89bf9e11-1400-4a4a-b474-f23baf5d0da5-istio-podinfo" pod "89bf9e11-1400-4a4a-b474-f23baf5d0da5" (UID: "89bf9e11-1400-4a4a-b474-f23baf5d0da5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:15:32.270687 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.270665    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14b0b7de-c4ed-4997-ba30-72f4d501786b-kube-api-access-cpddv" pod "14b0b7de-c4ed-4997-ba30-72f4d501786b" (UID: "14b0b7de-c4ed-4997-ba30-72f4d501786b"). InnerVolumeSpecName "kube-api-access-cpddv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:32.271056 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.271039    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89bf9e11-1400-4a4a-b474-f23baf5d0da5-istio-envoy" pod "89bf9e11-1400-4a4a-b474-f23baf5d0da5" (UID: "89bf9e11-1400-4a4a-b474-f23baf5d0da5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:32.287918 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.287871    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-1720c00c-549d8c448c-v5z86" podUID="c06ba561-05ef-49c8-b410-6aa8cb926b93" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:15:32.288123 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.288096    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-1720c00c-7f49c64bcf-pmktn" podUID="e2ad7926-76d5-48b1-8311-d3adb5742fd2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:15:32.367201 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.367109    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89bf9e11-1400-4a4a-b474-f23baf5d0da5-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:32.367201 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.367151    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14b0b7de-c4ed-4997-ba30-72f4d501786b-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:32.367201 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.367166    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89bf9e11-1400-4a4a-b474-f23baf5d0da5-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:32.367201 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.367178    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14b0b7de-c4ed-4997-ba30-72f4d501786b-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:32.367201 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.367191    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89bf9e11-1400-4a4a-b474-f23baf5d0da5-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:32.367201 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.367205    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpddv\" (UniqueName: \"kubernetes.io/projected/14b0b7de-c4ed-4997-ba30-72f4d501786b-kube-api-access-cpddv\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:32.367693 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.367220    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89bf9e11-1400-4a4a-b474-f23baf5d0da5-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:32.367693 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.367235    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89bf9e11-1400-4a4a-b474-f23baf5d0da5-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:32.367693 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.367248    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14b0b7de-c4ed-4997-ba30-72f4d501786b-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:32.367693 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.367259    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14b0b7de-c4ed-4997-ba30-72f4d501786b-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:32.367693 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.367271    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89bf9e11-1400-4a4a-b474-f23baf5d0da5-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:32.367693 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.367284    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5mv7\" (UniqueName: \"kubernetes.io/projected/89bf9e11-1400-4a4a-b474-f23baf5d0da5-kube-api-access-h5mv7\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:32.367693 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.367296    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89bf9e11-1400-4a4a-b474-f23baf5d0da5-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:32.367693 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.367308    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14b0b7de-c4ed-4997-ba30-72f4d501786b-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:32.367693 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.367320    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14b0b7de-c4ed-4997-ba30-72f4d501786b-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:32.367693 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.367333    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b0b7de-c4ed-4997-ba30-72f4d501786b-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:32.469108 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.469077    5315 generic.go:356] "Generic (PLEG): container finished" podID="14b0b7de-c4ed-4997-ba30-72f4d501786b" containerID="a59b1240a3cb08951e1a557a8931409d4f8cd084efa072830ce65869abaef681" exitCode=0
Nov 03 18:15:32.469270 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.469141    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb"
Nov 03 18:15:32.469270 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.469175    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb" event={"ID":"14b0b7de-c4ed-4997-ba30-72f4d501786b","Type":"ContainerDied","Data":"a59b1240a3cb08951e1a557a8931409d4f8cd084efa072830ce65869abaef681"}
Nov 03 18:15:32.469270 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.469238    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb" event={"ID":"14b0b7de-c4ed-4997-ba30-72f4d501786b","Type":"ContainerDied","Data":"547d27cdc78aa1c3e4d9d20e9575645198b3d316f93e45599dd37b59a4218e7b"}
Nov 03 18:15:32.469270 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.469254    5315 scope.go:122] "RemoveContainer" containerID="a59b1240a3cb08951e1a557a8931409d4f8cd084efa072830ce65869abaef681"
Nov 03 18:15:32.471300 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.471276    5315 generic.go:356] "Generic (PLEG): container finished" podID="89bf9e11-1400-4a4a-b474-f23baf5d0da5" containerID="532430e4867c14237c848ad891389fbccbec3c9d4ee034602f19f0bc0d05ee65" exitCode=0
Nov 03 18:15:32.471477 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.471461    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6" event={"ID":"89bf9e11-1400-4a4a-b474-f23baf5d0da5","Type":"ContainerDied","Data":"532430e4867c14237c848ad891389fbccbec3c9d4ee034602f19f0bc0d05ee65"}
Nov 03 18:15:32.471567 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.471504    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6" event={"ID":"89bf9e11-1400-4a4a-b474-f23baf5d0da5","Type":"ContainerDied","Data":"3218dcdb99b491de999a1748c5ad6e6b1e8c087c8b8a370d1d59f82ed7b151eb"}
Nov 03 18:15:32.471643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.471579    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6"
Nov 03 18:15:32.489168 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.489139    5315 scope.go:122] "RemoveContainer" containerID="a59b1240a3cb08951e1a557a8931409d4f8cd084efa072830ce65869abaef681"
Nov 03 18:15:32.490047 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:15:32.489605    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a59b1240a3cb08951e1a557a8931409d4f8cd084efa072830ce65869abaef681\": container with ID starting with a59b1240a3cb08951e1a557a8931409d4f8cd084efa072830ce65869abaef681 not found: ID does not exist" containerID="a59b1240a3cb08951e1a557a8931409d4f8cd084efa072830ce65869abaef681"
Nov 03 18:15:32.490164 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.490062    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a59b1240a3cb08951e1a557a8931409d4f8cd084efa072830ce65869abaef681"} err="failed to get container status \"a59b1240a3cb08951e1a557a8931409d4f8cd084efa072830ce65869abaef681\": rpc error: code = NotFound desc = could not find container \"a59b1240a3cb08951e1a557a8931409d4f8cd084efa072830ce65869abaef681\": container with ID starting with a59b1240a3cb08951e1a557a8931409d4f8cd084efa072830ce65869abaef681 not found: ID does not exist"
Nov 03 18:15:32.490164 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.490088    5315 scope.go:122] "RemoveContainer" containerID="532430e4867c14237c848ad891389fbccbec3c9d4ee034602f19f0bc0d05ee65"
Nov 03 18:15:32.504019 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.503437    5315 scope.go:122] "RemoveContainer" containerID="532430e4867c14237c848ad891389fbccbec3c9d4ee034602f19f0bc0d05ee65"
Nov 03 18:15:32.504378 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:15:32.504254    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"532430e4867c14237c848ad891389fbccbec3c9d4ee034602f19f0bc0d05ee65\": container with ID starting with 532430e4867c14237c848ad891389fbccbec3c9d4ee034602f19f0bc0d05ee65 not found: ID does not exist" containerID="532430e4867c14237c848ad891389fbccbec3c9d4ee034602f19f0bc0d05ee65"
Nov 03 18:15:32.504378 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.504301    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"532430e4867c14237c848ad891389fbccbec3c9d4ee034602f19f0bc0d05ee65"} err="failed to get container status \"532430e4867c14237c848ad891389fbccbec3c9d4ee034602f19f0bc0d05ee65\": rpc error: code = NotFound desc = could not find container \"532430e4867c14237c848ad891389fbccbec3c9d4ee034602f19f0bc0d05ee65\": container with ID starting with 532430e4867c14237c848ad891389fbccbec3c9d4ee034602f19f0bc0d05ee65 not found: ID does not exist"
Nov 03 18:15:32.686184 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.686098    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb"]
Nov 03 18:15:32.691433 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.691396    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-pwlpb"]
Nov 03 18:15:32.816660 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.807661    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6"]
Nov 03 18:15:32.833645 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.822516    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-fv6l6"]
Nov 03 18:15:32.870831 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.863623    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcf6b319-5txp6"]
Nov 03 18:15:32.870831 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.864379    5315 cpu_manager.go:414] "containerMap: removing container" podUID="e31a2c86-cff2-49ed-8f40-4020dfef4808" containerName="cleanup"
Nov 03 18:15:32.870831 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.864393    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e31a2c86-cff2-49ed-8f40-4020dfef4808" containerName="cleanup"
Nov 03 18:15:32.870831 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.864412    5315 cpu_manager.go:414] "containerMap: removing container" podUID="8ac59539-8ac1-466e-8d1b-b2be4c42c708" containerName="main"
Nov 03 18:15:32.870831 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.864419    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ac59539-8ac1-466e-8d1b-b2be4c42c708" containerName="main"
Nov 03 18:15:32.870831 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.864433    5315 cpu_manager.go:414] "containerMap: removing container" podUID="e31a2c86-cff2-49ed-8f40-4020dfef4808" containerName="cleanup"
Nov 03 18:15:32.870831 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.864442    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e31a2c86-cff2-49ed-8f40-4020dfef4808" containerName="cleanup"
Nov 03 18:15:32.870831 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.864455    5315 cpu_manager.go:414] "containerMap: removing container" podUID="e31a2c86-cff2-49ed-8f40-4020dfef4808" containerName="cleanup"
Nov 03 18:15:32.870831 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.864462    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e31a2c86-cff2-49ed-8f40-4020dfef4808" containerName="cleanup"
Nov 03 18:15:32.870831 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.864477    5315 cpu_manager.go:414] "containerMap: removing container" podUID="8d8504d9-e3e9-4542-8e76-3dbd404ed89f" containerName="cleanup"
Nov 03 18:15:32.870831 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.864484    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d8504d9-e3e9-4542-8e76-3dbd404ed89f" containerName="cleanup"
Nov 03 18:15:32.870831 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.864494    5315 cpu_manager.go:414] "containerMap: removing container" podUID="3d07e31d-20b5-47a1-901c-915565686286" containerName="istio-proxy"
Nov 03 18:15:32.870831 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.864502    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d07e31d-20b5-47a1-901c-915565686286" containerName="istio-proxy"
Nov 03 18:15:32.870831 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.864515    5315 cpu_manager.go:414] "containerMap: removing container" podUID="5d04a4ae-44e9-4f14-ada5-855c1efe7539" containerName="cleanup"
Nov 03 18:15:32.870831 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.864522    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d04a4ae-44e9-4f14-ada5-855c1efe7539" containerName="cleanup"
Nov 03 18:15:32.870831 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.864533    5315 cpu_manager.go:414] "containerMap: removing container" podUID="df8e0b71-e876-4cea-a7ae-3532c887c660" containerName="main"
Nov 03 18:15:32.870831 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.864540    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df8e0b71-e876-4cea-a7ae-3532c887c660" containerName="main"
Nov 03 18:15:32.870831 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.864559    5315 cpu_manager.go:414] "containerMap: removing container" podUID="26d8c8d7-89c0-47af-b90c-8b0677f71886" containerName="cleanup"
Nov 03 18:15:32.870831 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.864566    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26d8c8d7-89c0-47af-b90c-8b0677f71886" containerName="cleanup"
Nov 03 18:15:32.870831 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.864581    5315 cpu_manager.go:414] "containerMap: removing container" podUID="5d04a4ae-44e9-4f14-ada5-855c1efe7539" containerName="cleanup"
Nov 03 18:15:32.870831 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.864588    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d04a4ae-44e9-4f14-ada5-855c1efe7539" containerName="cleanup"
Nov 03 18:15:32.870831 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.864597    5315 cpu_manager.go:414] "containerMap: removing container" podUID="14b0b7de-c4ed-4997-ba30-72f4d501786b" containerName="istio-proxy"
Nov 03 18:15:32.872205 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.864605    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14b0b7de-c4ed-4997-ba30-72f4d501786b" containerName="istio-proxy"
Nov 03 18:15:32.872395 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.872307    5315 cpu_manager.go:414] "containerMap: removing container" podUID="26d8c8d7-89c0-47af-b90c-8b0677f71886" containerName="cleanup"
Nov 03 18:15:32.872395 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.872323    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26d8c8d7-89c0-47af-b90c-8b0677f71886" containerName="cleanup"
Nov 03 18:15:32.872703 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.872659    5315 cpu_manager.go:414] "containerMap: removing container" podUID="54bd478e-0685-4ec9-8aa0-5c3bd73c56c2" containerName="cleanup"
Nov 03 18:15:32.872703 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.872676    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54bd478e-0685-4ec9-8aa0-5c3bd73c56c2" containerName="cleanup"
Nov 03 18:15:32.872933 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.872857    5315 cpu_manager.go:414] "containerMap: removing container" podUID="26d8c8d7-89c0-47af-b90c-8b0677f71886" containerName="cleanup"
Nov 03 18:15:32.872933 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.872868    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26d8c8d7-89c0-47af-b90c-8b0677f71886" containerName="cleanup"
Nov 03 18:15:32.872933 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.872889    5315 cpu_manager.go:414] "containerMap: removing container" podUID="8d8504d9-e3e9-4542-8e76-3dbd404ed89f" containerName="cleanup"
Nov 03 18:15:32.872933 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.872895    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d8504d9-e3e9-4542-8e76-3dbd404ed89f" containerName="cleanup"
Nov 03 18:15:32.873260 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.872916    5315 cpu_manager.go:414] "containerMap: removing container" podUID="daf3798f-da45-4287-94f6-eb32897e807b" containerName="istio-proxy"
Nov 03 18:15:32.873260 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.873189    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daf3798f-da45-4287-94f6-eb32897e807b" containerName="istio-proxy"
Nov 03 18:15:32.873260 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.873200    5315 cpu_manager.go:414] "containerMap: removing container" podUID="89bf9e11-1400-4a4a-b474-f23baf5d0da5" containerName="istio-proxy"
Nov 03 18:15:32.873260 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.873207    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89bf9e11-1400-4a4a-b474-f23baf5d0da5" containerName="istio-proxy"
Nov 03 18:15:32.873260 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.873217    5315 cpu_manager.go:414] "containerMap: removing container" podUID="54bd478e-0685-4ec9-8aa0-5c3bd73c56c2" containerName="cleanup"
Nov 03 18:15:32.873260 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.873224    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54bd478e-0685-4ec9-8aa0-5c3bd73c56c2" containerName="cleanup"
Nov 03 18:15:32.883100 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.881151    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d04a4ae-44e9-4f14-ada5-855c1efe7539" containerName="cleanup"
Nov 03 18:15:32.883100 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.881184    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="e31a2c86-cff2-49ed-8f40-4020dfef4808" containerName="cleanup"
Nov 03 18:15:32.883100 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.881198    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ac59539-8ac1-466e-8d1b-b2be4c42c708" containerName="main"
Nov 03 18:15:32.883100 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.881207    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="e31a2c86-cff2-49ed-8f40-4020dfef4808" containerName="cleanup"
Nov 03 18:15:32.883100 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.881217    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="e31a2c86-cff2-49ed-8f40-4020dfef4808" containerName="cleanup"
Nov 03 18:15:32.883100 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.881230    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d8504d9-e3e9-4542-8e76-3dbd404ed89f" containerName="cleanup"
Nov 03 18:15:32.883100 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.881239    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d07e31d-20b5-47a1-901c-915565686286" containerName="istio-proxy"
Nov 03 18:15:32.883100 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.881247    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d04a4ae-44e9-4f14-ada5-855c1efe7539" containerName="cleanup"
Nov 03 18:15:32.883100 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.881255    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="df8e0b71-e876-4cea-a7ae-3532c887c660" containerName="main"
Nov 03 18:15:32.883100 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.881266    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="26d8c8d7-89c0-47af-b90c-8b0677f71886" containerName="cleanup"
Nov 03 18:15:32.883100 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.881276    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="14b0b7de-c4ed-4997-ba30-72f4d501786b" containerName="istio-proxy"
Nov 03 18:15:32.883100 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.881285    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="26d8c8d7-89c0-47af-b90c-8b0677f71886" containerName="cleanup"
Nov 03 18:15:32.883100 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.881294    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d8504d9-e3e9-4542-8e76-3dbd404ed89f" containerName="cleanup"
Nov 03 18:15:32.883100 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.881302    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="54bd478e-0685-4ec9-8aa0-5c3bd73c56c2" containerName="cleanup"
Nov 03 18:15:32.883100 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.881316    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="daf3798f-da45-4287-94f6-eb32897e807b" containerName="istio-proxy"
Nov 03 18:15:32.883100 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.881325    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="89bf9e11-1400-4a4a-b474-f23baf5d0da5" containerName="istio-proxy"
Nov 03 18:15:32.883100 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.881335    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="54bd478e-0685-4ec9-8aa0-5c3bd73c56c2" containerName="cleanup"
Nov 03 18:15:32.889279 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.888667    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcf6b319-5txp6"
Nov 03 18:15:32.931512 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.931476    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcf6b319-5txp6"]
Nov 03 18:15:32.938084 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.938009    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:15:32.938270 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.938020    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:15:32.938397 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.938272    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:15:32.938589 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:32.938570    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:15:33.007415 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:33.004453    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f66eb4b-1a62-4dad-b70a-8fdd8f715b70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcf6b319-5txp6\" (UID: \"0f66eb4b-1a62-4dad-b70a-8fdd8f715b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcf6b319-5txp6"
Nov 03 18:15:33.007415 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:33.004748    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f66eb4b-1a62-4dad-b70a-8fdd8f715b70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcf6b319-5txp6\" (UID: \"0f66eb4b-1a62-4dad-b70a-8fdd8f715b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcf6b319-5txp6"
Nov 03 18:15:33.007415 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:33.004855    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6g4c\" (UniqueName: \"kubernetes.io/projected/0f66eb4b-1a62-4dad-b70a-8fdd8f715b70-kube-api-access-w6g4c\") pod \"maas-api-revoke-sub-bcf6b319-5txp6\" (UID: \"0f66eb4b-1a62-4dad-b70a-8fdd8f715b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcf6b319-5txp6"
Nov 03 18:15:33.106749 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:33.106706    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14b0b7de-c4ed-4997-ba30-72f4d501786b" path="/var/lib/kubelet/pods/14b0b7de-c4ed-4997-ba30-72f4d501786b/volumes"
Nov 03 18:15:33.106749 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:33.106726    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f66eb4b-1a62-4dad-b70a-8fdd8f715b70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcf6b319-5txp6\" (UID: \"0f66eb4b-1a62-4dad-b70a-8fdd8f715b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcf6b319-5txp6"
Nov 03 18:15:33.107021 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:33.106776    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6g4c\" (UniqueName: \"kubernetes.io/projected/0f66eb4b-1a62-4dad-b70a-8fdd8f715b70-kube-api-access-w6g4c\") pod \"maas-api-revoke-sub-bcf6b319-5txp6\" (UID: \"0f66eb4b-1a62-4dad-b70a-8fdd8f715b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcf6b319-5txp6"
Nov 03 18:15:33.107021 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:33.106844    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f66eb4b-1a62-4dad-b70a-8fdd8f715b70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcf6b319-5txp6\" (UID: \"0f66eb4b-1a62-4dad-b70a-8fdd8f715b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcf6b319-5txp6"
Nov 03 18:15:33.107688 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:33.107660    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f66eb4b-1a62-4dad-b70a-8fdd8f715b70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcf6b319-5txp6\" (UID: \"0f66eb4b-1a62-4dad-b70a-8fdd8f715b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcf6b319-5txp6"
Nov 03 18:15:33.108059 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:33.108038    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89bf9e11-1400-4a4a-b474-f23baf5d0da5" path="/var/lib/kubelet/pods/89bf9e11-1400-4a4a-b474-f23baf5d0da5/volumes"
Nov 03 18:15:33.109186 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:33.109158    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f66eb4b-1a62-4dad-b70a-8fdd8f715b70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcf6b319-5txp6\" (UID: \"0f66eb4b-1a62-4dad-b70a-8fdd8f715b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcf6b319-5txp6"
Nov 03 18:15:33.164571 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:33.164538    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6g4c\" (UniqueName: \"kubernetes.io/projected/0f66eb4b-1a62-4dad-b70a-8fdd8f715b70-kube-api-access-w6g4c\") pod \"maas-api-revoke-sub-bcf6b319-5txp6\" (UID: \"0f66eb4b-1a62-4dad-b70a-8fdd8f715b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcf6b319-5txp6"
Nov 03 18:15:33.222727 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:33.222640    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcf6b319-5txp6"
Nov 03 18:15:33.664672 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:33.664083    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcf6b319-5txp6"]
Nov 03 18:15:33.665646 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:15:33.665576    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f66eb4b_1a62_4dad_b70a_8fdd8f715b70.slice/crio-8b8887416b1022c6f756ef15395e69ed2f2c5f33a61a4ab9b8579287153992d6 WatchSource:0}: Error finding container 8b8887416b1022c6f756ef15395e69ed2f2c5f33a61a4ab9b8579287153992d6: Status 404 returned error can't find the container with id 8b8887416b1022c6f756ef15395e69ed2f2c5f33a61a4ab9b8579287153992d6
Nov 03 18:15:34.490429 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:34.490339    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcf6b319-5txp6" event={"ID":"0f66eb4b-1a62-4dad-b70a-8fdd8f715b70","Type":"ContainerStarted","Data":"12f308d10ef3d58e4bf4bab76a79145a186ecf2b8beb5c79f8c9b3bb7867aa02"}
Nov 03 18:15:34.490429 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:34.490399    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcf6b319-5txp6" event={"ID":"0f66eb4b-1a62-4dad-b70a-8fdd8f715b70","Type":"ContainerStarted","Data":"8b8887416b1022c6f756ef15395e69ed2f2c5f33a61a4ab9b8579287153992d6"}
Nov 03 18:15:34.609631 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:34.606181    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcf6b319-5txp6" podStartSLOduration=2.606159758 podStartE2EDuration="2.606159758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:15:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:15:34.598887518 +0000 UTC m=+1773.615085587" watchObservedRunningTime="2026-11-03 18:15:34.606159758 +0000 UTC m=+1773.622357828"
Nov 03 18:15:34.621924 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:34.621879    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-a86228-f656db785-jcvfr" podUID="6434783b-1272-4ccd-b5c4-5a7aeeab58c7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:15:35.499329 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:35.499285    5315 generic.go:356] "Generic (PLEG): container finished" podID="0f66eb4b-1a62-4dad-b70a-8fdd8f715b70" containerID="12f308d10ef3d58e4bf4bab76a79145a186ecf2b8beb5c79f8c9b3bb7867aa02" exitCode=0
Nov 03 18:15:35.499329 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:35.499332    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcf6b319-5txp6" event={"ID":"0f66eb4b-1a62-4dad-b70a-8fdd8f715b70","Type":"ContainerDied","Data":"12f308d10ef3d58e4bf4bab76a79145a186ecf2b8beb5c79f8c9b3bb7867aa02"}
Nov 03 18:15:36.698172 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:36.698140    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcf6b319-5txp6"
Nov 03 18:15:36.774413 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:36.774330    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f66eb4b-1a62-4dad-b70a-8fdd8f715b70-kube-api-access-w6g4c\" (UniqueName: \"kubernetes.io/projected/0f66eb4b-1a62-4dad-b70a-8fdd8f715b70-kube-api-access-w6g4c\") pod \"0f66eb4b-1a62-4dad-b70a-8fdd8f715b70\" (UID: \"0f66eb4b-1a62-4dad-b70a-8fdd8f715b70\") "
Nov 03 18:15:36.774573 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:36.774430    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f66eb4b-1a62-4dad-b70a-8fdd8f715b70-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f66eb4b-1a62-4dad-b70a-8fdd8f715b70-maas-api-openshift-ca\") pod \"0f66eb4b-1a62-4dad-b70a-8fdd8f715b70\" (UID: \"0f66eb4b-1a62-4dad-b70a-8fdd8f715b70\") "
Nov 03 18:15:36.774573 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:36.774492    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f66eb4b-1a62-4dad-b70a-8fdd8f715b70-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f66eb4b-1a62-4dad-b70a-8fdd8f715b70-maas-api-xks-ca\") pod \"0f66eb4b-1a62-4dad-b70a-8fdd8f715b70\" (UID: \"0f66eb4b-1a62-4dad-b70a-8fdd8f715b70\") "
Nov 03 18:15:36.774907 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:36.774868    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f66eb4b-1a62-4dad-b70a-8fdd8f715b70-maas-api-openshift-ca" pod "0f66eb4b-1a62-4dad-b70a-8fdd8f715b70" (UID: "0f66eb4b-1a62-4dad-b70a-8fdd8f715b70"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:36.775143 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:36.775123    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f66eb4b-1a62-4dad-b70a-8fdd8f715b70-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:36.776460 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:36.776437    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f66eb4b-1a62-4dad-b70a-8fdd8f715b70-kube-api-access-w6g4c" pod "0f66eb4b-1a62-4dad-b70a-8fdd8f715b70" (UID: "0f66eb4b-1a62-4dad-b70a-8fdd8f715b70"). InnerVolumeSpecName "kube-api-access-w6g4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:36.778015 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:36.777983    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f66eb4b-1a62-4dad-b70a-8fdd8f715b70-maas-api-xks-ca" pod "0f66eb4b-1a62-4dad-b70a-8fdd8f715b70" (UID: "0f66eb4b-1a62-4dad-b70a-8fdd8f715b70"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:15:36.876546 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:36.876506    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f66eb4b-1a62-4dad-b70a-8fdd8f715b70-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:36.876546 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:36.876540    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6g4c\" (UniqueName: \"kubernetes.io/projected/0f66eb4b-1a62-4dad-b70a-8fdd8f715b70-kube-api-access-w6g4c\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:37.514642 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:37.514588    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcf6b319-5txp6" event={"ID":"0f66eb4b-1a62-4dad-b70a-8fdd8f715b70","Type":"ContainerDied","Data":"8b8887416b1022c6f756ef15395e69ed2f2c5f33a61a4ab9b8579287153992d6"}
Nov 03 18:15:37.514642 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:37.514627    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcf6b319-5txp6"
Nov 03 18:15:37.515821 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:37.514652    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b8887416b1022c6f756ef15395e69ed2f2c5f33a61a4ab9b8579287153992d6"
Nov 03 18:15:38.522470 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:38.522436    5315 generic.go:356] "Generic (PLEG): container finished" podID="915e71a9-f7c9-4be4-929a-af6be579b6bd" containerID="8075f30f685a50373bf0aad4c0e226ba8e9fae327819f934e88dc058ed053ce1" exitCode=0
Nov 03 18:15:38.522952 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:38.522505    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g" event={"ID":"915e71a9-f7c9-4be4-929a-af6be579b6bd","Type":"ContainerDied","Data":"8075f30f685a50373bf0aad4c0e226ba8e9fae327819f934e88dc058ed053ce1"}
Nov 03 18:15:38.523015 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:38.522982    5315 scope.go:122] "RemoveContainer" containerID="8075f30f685a50373bf0aad4c0e226ba8e9fae327819f934e88dc058ed053ce1"
Nov 03 18:15:38.524635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:38.524576    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:15:38.524714 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:38.524691    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:15:38.524752 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:38.524739    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:15:39.557708 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:39.557671    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g" event={"ID":"915e71a9-f7c9-4be4-929a-af6be579b6bd","Type":"ContainerStarted","Data":"8f4337e2ad655e0367f10693fec2249a1c5648782bc150d4bae14cad4d970281"}
Nov 03 18:15:40.010362 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:40.010332    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-a86228-f656db785-jcvfr"
Nov 03 18:15:40.131232 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:40.130054    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6434783b-1272-4ccd-b5c4-5a7aeeab58c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6434783b-1272-4ccd-b5c4-5a7aeeab58c7-plugins-config-volume\") pod \"6434783b-1272-4ccd-b5c4-5a7aeeab58c7\" (UID: \"6434783b-1272-4ccd-b5c4-5a7aeeab58c7\") "
Nov 03 18:15:40.131232 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:40.130324    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6434783b-1272-4ccd-b5c4-5a7aeeab58c7-kube-api-access-2cf6q\" (UniqueName: \"kubernetes.io/projected/6434783b-1272-4ccd-b5c4-5a7aeeab58c7-kube-api-access-2cf6q\") pod \"6434783b-1272-4ccd-b5c4-5a7aeeab58c7\" (UID: \"6434783b-1272-4ccd-b5c4-5a7aeeab58c7\") "
Nov 03 18:15:40.131993 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:40.131924    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6434783b-1272-4ccd-b5c4-5a7aeeab58c7-plugins-config-volume" pod "6434783b-1272-4ccd-b5c4-5a7aeeab58c7" (UID: "6434783b-1272-4ccd-b5c4-5a7aeeab58c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:40.157502 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:40.157448    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6434783b-1272-4ccd-b5c4-5a7aeeab58c7-kube-api-access-2cf6q" pod "6434783b-1272-4ccd-b5c4-5a7aeeab58c7" (UID: "6434783b-1272-4ccd-b5c4-5a7aeeab58c7"). InnerVolumeSpecName "kube-api-access-2cf6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:40.169072 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:40.163942    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g"
Nov 03 18:15:40.169072 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:40.164247    5315 patch_prober.go:29] interesting pod/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Nov 03 18:15:40.169072 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:40.164293    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g" podUID="915e71a9-f7c9-4be4-929a-af6be579b6bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Nov 03 18:15:40.237635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:40.236709    5315 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6434783b-1272-4ccd-b5c4-5a7aeeab58c7-plugins-config-volume\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:40.237635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:40.236755    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cf6q\" (UniqueName: \"kubernetes.io/projected/6434783b-1272-4ccd-b5c4-5a7aeeab58c7-kube-api-access-2cf6q\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:40.591664 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:40.591017    5315 generic.go:356] "Generic (PLEG): container finished" podID="6434783b-1272-4ccd-b5c4-5a7aeeab58c7" containerID="4f62f10b9e6389ae06381c234bdf4fa35ab045d5a47f6f0aef6e0e65fce8599e" exitCode=0
Nov 03 18:15:40.591664 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:40.591091    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-a86228-f656db785-jcvfr"
Nov 03 18:15:40.591664 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:40.591137    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-a86228-f656db785-jcvfr" event={"ID":"6434783b-1272-4ccd-b5c4-5a7aeeab58c7","Type":"ContainerDied","Data":"4f62f10b9e6389ae06381c234bdf4fa35ab045d5a47f6f0aef6e0e65fce8599e"}
Nov 03 18:15:40.591664 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:40.591191    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-a86228-f656db785-jcvfr" event={"ID":"6434783b-1272-4ccd-b5c4-5a7aeeab58c7","Type":"ContainerDied","Data":"ed6844c0ddc724170ee8a8bfff02c1402ed187e9e8752f51be95b0b062dc08f2"}
Nov 03 18:15:40.591664 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:40.591228    5315 scope.go:122] "RemoveContainer" containerID="4f62f10b9e6389ae06381c234bdf4fa35ab045d5a47f6f0aef6e0e65fce8599e"
Nov 03 18:15:40.609116 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:40.609079    5315 scope.go:122] "RemoveContainer" containerID="4f62f10b9e6389ae06381c234bdf4fa35ab045d5a47f6f0aef6e0e65fce8599e"
Nov 03 18:15:40.609468 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:15:40.609443    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f62f10b9e6389ae06381c234bdf4fa35ab045d5a47f6f0aef6e0e65fce8599e\": container with ID starting with 4f62f10b9e6389ae06381c234bdf4fa35ab045d5a47f6f0aef6e0e65fce8599e not found: ID does not exist" containerID="4f62f10b9e6389ae06381c234bdf4fa35ab045d5a47f6f0aef6e0e65fce8599e"
Nov 03 18:15:40.609543 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:40.609488    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f62f10b9e6389ae06381c234bdf4fa35ab045d5a47f6f0aef6e0e65fce8599e"} err="failed to get container status \"4f62f10b9e6389ae06381c234bdf4fa35ab045d5a47f6f0aef6e0e65fce8599e\": rpc error: code = NotFound desc = could not find container \"4f62f10b9e6389ae06381c234bdf4fa35ab045d5a47f6f0aef6e0e65fce8599e\": container with ID starting with 4f62f10b9e6389ae06381c234bdf4fa35ab045d5a47f6f0aef6e0e65fce8599e not found: ID does not exist"
Nov 03 18:15:40.752743 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:40.752698    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-a86228-f656db785-jcvfr"]
Nov 03 18:15:40.798278 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:40.798238    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-a86228-f656db785-jcvfr"]
Nov 03 18:15:41.106442 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:41.106408    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6434783b-1272-4ccd-b5c4-5a7aeeab58c7" path="/var/lib/kubelet/pods/6434783b-1272-4ccd-b5c4-5a7aeeab58c7/volumes"
Nov 03 18:15:41.164096 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:41.164056    5315 patch_prober.go:29] interesting pod/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Nov 03 18:15:41.164271 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:41.164111    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g" podUID="915e71a9-f7c9-4be4-929a-af6be579b6bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Nov 03 18:15:42.164341 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:42.164292    5315 patch_prober.go:29] interesting pod/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Nov 03 18:15:42.164805 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:42.164368    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g" podUID="915e71a9-f7c9-4be4-929a-af6be579b6bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Nov 03 18:15:42.288036 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:42.287992    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-1720c00c-549d8c448c-v5z86" podUID="c06ba561-05ef-49c8-b410-6aa8cb926b93" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:15:42.288211 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:42.288070    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-1720c00c-7f49c64bcf-pmktn" podUID="e2ad7926-76d5-48b1-8311-d3adb5742fd2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:15:43.163835 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:43.163797    5315 patch_prober.go:29] interesting pod/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Nov 03 18:15:43.164048 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:43.163875    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g" podUID="915e71a9-f7c9-4be4-929a-af6be579b6bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Nov 03 18:15:43.653852 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:43.653820    5315 generic.go:356] "Generic (PLEG): container finished" podID="7296122e-d510-4826-9a6a-2f663d6603bf" containerID="4defaaf0656c36b5a0a57ca43a2b83dd12eaff89e5a3fe5d8ca68d955cf57dd1" exitCode=0
Nov 03 18:15:43.654303 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:43.653898    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56" event={"ID":"7296122e-d510-4826-9a6a-2f663d6603bf","Type":"ContainerDied","Data":"4defaaf0656c36b5a0a57ca43a2b83dd12eaff89e5a3fe5d8ca68d955cf57dd1"}
Nov 03 18:15:43.654353 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:43.654344    5315 scope.go:122] "RemoveContainer" containerID="4defaaf0656c36b5a0a57ca43a2b83dd12eaff89e5a3fe5d8ca68d955cf57dd1"
Nov 03 18:15:43.655800 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:43.655772    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:15:43.655867 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:43.655841    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:15:43.655905 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:43.655875    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:15:44.164506 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:44.164470    5315 patch_prober.go:29] interesting pod/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Nov 03 18:15:44.164700 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:44.164526    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g" podUID="915e71a9-f7c9-4be4-929a-af6be579b6bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Nov 03 18:15:44.663902 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:44.663859    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56" event={"ID":"7296122e-d510-4826-9a6a-2f663d6603bf","Type":"ContainerStarted","Data":"fa2f6550978008e98297621aacbd87dab09e8452943faa71a97fb27069e9e68d"}
Nov 03 18:15:45.168542 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:45.168509    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g"
Nov 03 18:15:45.168973 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:45.168944    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g"
Nov 03 18:15:45.169551 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:45.169531    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g"
Nov 03 18:15:45.428096 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:45.428011    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56"
Nov 03 18:15:45.428416 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:45.428389    5315 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Nov 03 18:15:45.428569 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:45.428544    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56" podUID="7296122e-d510-4826-9a6a-2f663d6603bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Nov 03 18:15:46.447646 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:46.447283    5315 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Nov 03 18:15:46.447646 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:46.447354    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56" podUID="7296122e-d510-4826-9a6a-2f663d6603bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Nov 03 18:15:47.365282 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:47.365249    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1720c00c-7f49c64bcf-pmktn"
Nov 03 18:15:47.444634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:47.444008    5315 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Nov 03 18:15:47.444634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:47.444073    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56" podUID="7296122e-d510-4826-9a6a-2f663d6603bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Nov 03 18:15:47.495653 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:47.493726    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2ad7926-76d5-48b1-8311-d3adb5742fd2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2ad7926-76d5-48b1-8311-d3adb5742fd2-plugins-config-volume\") pod \"e2ad7926-76d5-48b1-8311-d3adb5742fd2\" (UID: \"e2ad7926-76d5-48b1-8311-d3adb5742fd2\") "
Nov 03 18:15:47.495653 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:47.493808    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2ad7926-76d5-48b1-8311-d3adb5742fd2-kube-api-access-pxtlc\" (UniqueName: \"kubernetes.io/projected/e2ad7926-76d5-48b1-8311-d3adb5742fd2-kube-api-access-pxtlc\") pod \"e2ad7926-76d5-48b1-8311-d3adb5742fd2\" (UID: \"e2ad7926-76d5-48b1-8311-d3adb5742fd2\") "
Nov 03 18:15:47.514400 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:47.514355    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2ad7926-76d5-48b1-8311-d3adb5742fd2-plugins-config-volume" pod "e2ad7926-76d5-48b1-8311-d3adb5742fd2" (UID: "e2ad7926-76d5-48b1-8311-d3adb5742fd2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:47.518734 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:47.518682    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2ad7926-76d5-48b1-8311-d3adb5742fd2-kube-api-access-pxtlc" pod "e2ad7926-76d5-48b1-8311-d3adb5742fd2" (UID: "e2ad7926-76d5-48b1-8311-d3adb5742fd2"). InnerVolumeSpecName "kube-api-access-pxtlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:47.587674 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:47.587595    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1720c00c-549d8c448c-v5z86"
Nov 03 18:15:47.596378 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:47.596348    5315 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2ad7926-76d5-48b1-8311-d3adb5742fd2-plugins-config-volume\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:47.596378 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:47.596375    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxtlc\" (UniqueName: \"kubernetes.io/projected/e2ad7926-76d5-48b1-8311-d3adb5742fd2-kube-api-access-pxtlc\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:47.687182 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:47.687137    5315 generic.go:356] "Generic (PLEG): container finished" podID="c06ba561-05ef-49c8-b410-6aa8cb926b93" containerID="898f7e8995a9ba67aeaa853d027be769612a9ecbabf900a8efbf59446f782763" exitCode=0
Nov 03 18:15:47.687385 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:47.687199    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1720c00c-549d8c448c-v5z86"
Nov 03 18:15:47.687385 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:47.687227    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1720c00c-549d8c448c-v5z86" event={"ID":"c06ba561-05ef-49c8-b410-6aa8cb926b93","Type":"ContainerDied","Data":"898f7e8995a9ba67aeaa853d027be769612a9ecbabf900a8efbf59446f782763"}
Nov 03 18:15:47.687385 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:47.687282    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1720c00c-549d8c448c-v5z86" event={"ID":"c06ba561-05ef-49c8-b410-6aa8cb926b93","Type":"ContainerDied","Data":"c585cfa1866edeb9c7194a1bf31f970de585c0c4fb0759baa203b5964106961d"}
Nov 03 18:15:47.687385 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:47.687318    5315 scope.go:122] "RemoveContainer" containerID="898f7e8995a9ba67aeaa853d027be769612a9ecbabf900a8efbf59446f782763"
Nov 03 18:15:47.688814 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:47.688791    5315 generic.go:356] "Generic (PLEG): container finished" podID="e2ad7926-76d5-48b1-8311-d3adb5742fd2" containerID="0aad1bbb8c55382336f0f49a62296ccd8b182596d6bd82407766e0ba6afe901f" exitCode=0
Nov 03 18:15:47.688941 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:47.688840    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1720c00c-7f49c64bcf-pmktn" event={"ID":"e2ad7926-76d5-48b1-8311-d3adb5742fd2","Type":"ContainerDied","Data":"0aad1bbb8c55382336f0f49a62296ccd8b182596d6bd82407766e0ba6afe901f"}
Nov 03 18:15:47.688941 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:47.688853    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1720c00c-7f49c64bcf-pmktn"
Nov 03 18:15:47.688941 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:47.688882    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1720c00c-7f49c64bcf-pmktn" event={"ID":"e2ad7926-76d5-48b1-8311-d3adb5742fd2","Type":"ContainerDied","Data":"ef25a79792fb289dc3c1d9b08ad77e32351951906e6e037d8adee8bcf72fd133"}
Nov 03 18:15:47.696964 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:47.696933    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c06ba561-05ef-49c8-b410-6aa8cb926b93-kube-api-access-bwhcq\" (UniqueName: \"kubernetes.io/projected/c06ba561-05ef-49c8-b410-6aa8cb926b93-kube-api-access-bwhcq\") pod \"c06ba561-05ef-49c8-b410-6aa8cb926b93\" (UID: \"c06ba561-05ef-49c8-b410-6aa8cb926b93\") "
Nov 03 18:15:47.697111 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:47.697030    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c06ba561-05ef-49c8-b410-6aa8cb926b93-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c06ba561-05ef-49c8-b410-6aa8cb926b93-plugins-config-volume\") pod \"c06ba561-05ef-49c8-b410-6aa8cb926b93\" (UID: \"c06ba561-05ef-49c8-b410-6aa8cb926b93\") "
Nov 03 18:15:47.704384 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:47.700818    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c06ba561-05ef-49c8-b410-6aa8cb926b93-plugins-config-volume" pod "c06ba561-05ef-49c8-b410-6aa8cb926b93" (UID: "c06ba561-05ef-49c8-b410-6aa8cb926b93"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:47.706041 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:47.705978    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c06ba561-05ef-49c8-b410-6aa8cb926b93-kube-api-access-bwhcq" pod "c06ba561-05ef-49c8-b410-6aa8cb926b93" (UID: "c06ba561-05ef-49c8-b410-6aa8cb926b93"). InnerVolumeSpecName "kube-api-access-bwhcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:47.710594 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:47.710139    5315 scope.go:122] "RemoveContainer" containerID="898f7e8995a9ba67aeaa853d027be769612a9ecbabf900a8efbf59446f782763"
Nov 03 18:15:47.710594 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:15:47.710542    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"898f7e8995a9ba67aeaa853d027be769612a9ecbabf900a8efbf59446f782763\": container with ID starting with 898f7e8995a9ba67aeaa853d027be769612a9ecbabf900a8efbf59446f782763 not found: ID does not exist" containerID="898f7e8995a9ba67aeaa853d027be769612a9ecbabf900a8efbf59446f782763"
Nov 03 18:15:47.710594 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:47.710575    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"898f7e8995a9ba67aeaa853d027be769612a9ecbabf900a8efbf59446f782763"} err="failed to get container status \"898f7e8995a9ba67aeaa853d027be769612a9ecbabf900a8efbf59446f782763\": rpc error: code = NotFound desc = could not find container \"898f7e8995a9ba67aeaa853d027be769612a9ecbabf900a8efbf59446f782763\": container with ID starting with 898f7e8995a9ba67aeaa853d027be769612a9ecbabf900a8efbf59446f782763 not found: ID does not exist"
Nov 03 18:15:47.710594 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:47.710594    5315 scope.go:122] "RemoveContainer" containerID="0aad1bbb8c55382336f0f49a62296ccd8b182596d6bd82407766e0ba6afe901f"
Nov 03 18:15:47.740839 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:47.730815    5315 scope.go:122] "RemoveContainer" containerID="0aad1bbb8c55382336f0f49a62296ccd8b182596d6bd82407766e0ba6afe901f"
Nov 03 18:15:47.740839 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:15:47.731274    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0aad1bbb8c55382336f0f49a62296ccd8b182596d6bd82407766e0ba6afe901f\": container with ID starting with 0aad1bbb8c55382336f0f49a62296ccd8b182596d6bd82407766e0ba6afe901f not found: ID does not exist" containerID="0aad1bbb8c55382336f0f49a62296ccd8b182596d6bd82407766e0ba6afe901f"
Nov 03 18:15:47.740839 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:47.731311    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aad1bbb8c55382336f0f49a62296ccd8b182596d6bd82407766e0ba6afe901f"} err="failed to get container status \"0aad1bbb8c55382336f0f49a62296ccd8b182596d6bd82407766e0ba6afe901f\": rpc error: code = NotFound desc = could not find container \"0aad1bbb8c55382336f0f49a62296ccd8b182596d6bd82407766e0ba6afe901f\": container with ID starting with 0aad1bbb8c55382336f0f49a62296ccd8b182596d6bd82407766e0ba6afe901f not found: ID does not exist"
Nov 03 18:15:47.798633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:47.798581    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwhcq\" (UniqueName: \"kubernetes.io/projected/c06ba561-05ef-49c8-b410-6aa8cb926b93-kube-api-access-bwhcq\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:47.798633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:47.798626    5315 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c06ba561-05ef-49c8-b410-6aa8cb926b93-plugins-config-volume\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:47.906003 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:47.905907    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1720c00c-7f49c64bcf-pmktn"]
Nov 03 18:15:47.912104 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:47.912069    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1720c00c-7f49c64bcf-pmktn"]
Nov 03 18:15:48.115752 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:48.115717    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1720c00c-549d8c448c-v5z86"]
Nov 03 18:15:48.118469 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:48.118442    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1720c00c-549d8c448c-v5z86"]
Nov 03 18:15:48.425661 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:48.424054    5315 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Nov 03 18:15:48.425661 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:48.424133    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56" podUID="7296122e-d510-4826-9a6a-2f663d6603bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Nov 03 18:15:49.104384 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:49.104346    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c06ba561-05ef-49c8-b410-6aa8cb926b93" path="/var/lib/kubelet/pods/c06ba561-05ef-49c8-b410-6aa8cb926b93/volumes"
Nov 03 18:15:49.105057 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:49.105034    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2ad7926-76d5-48b1-8311-d3adb5742fd2" path="/var/lib/kubelet/pods/e2ad7926-76d5-48b1-8311-d3adb5742fd2/volumes"
Nov 03 18:15:49.423421 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:49.423336    5315 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Nov 03 18:15:49.423708 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:49.423680    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56" podUID="7296122e-d510-4826-9a6a-2f663d6603bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Nov 03 18:15:50.429317 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:50.429288    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56"
Nov 03 18:15:50.429758 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:50.429544    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56"
Nov 03 18:15:50.430252 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:50.430234    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56"
Nov 03 18:15:53.310634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:53.310313    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aab753-469nk"]
Nov 03 18:15:53.344388 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:53.344346    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aab753-469nk"]
Nov 03 18:15:54.300898 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:54.300858    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a86228-7nnxk"]
Nov 03 18:15:54.345634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:54.343106    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a86228-7nnxk"]
Nov 03 18:15:54.578510 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:15:54.578421    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:15:55.104393 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:55.104361    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51" path="/var/lib/kubelet/pods/658c1fb2-bc3b-4d9a-8e9a-e23c874b7b51/volumes"
Nov 03 18:15:55.104985 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:55.104960    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b15f8554-6d3e-46bf-886f-e32da257adcc" path="/var/lib/kubelet/pods/b15f8554-6d3e-46bf-886f-e32da257adcc/volumes"
Nov 03 18:15:56.597238 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:56.597196    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-80aca1-dmkj8"]
Nov 03 18:15:56.597891 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:56.597873    5315 cpu_manager.go:414] "containerMap: removing container" podUID="6434783b-1272-4ccd-b5c4-5a7aeeab58c7" containerName="payload-processing"
Nov 03 18:15:56.597941 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:56.597892    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6434783b-1272-4ccd-b5c4-5a7aeeab58c7" containerName="payload-processing"
Nov 03 18:15:56.597941 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:56.597907    5315 cpu_manager.go:414] "containerMap: removing container" podUID="54bd478e-0685-4ec9-8aa0-5c3bd73c56c2" containerName="cleanup"
Nov 03 18:15:56.597941 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:56.597912    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54bd478e-0685-4ec9-8aa0-5c3bd73c56c2" containerName="cleanup"
Nov 03 18:15:56.597941 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:56.597928    5315 cpu_manager.go:414] "containerMap: removing container" podUID="e2ad7926-76d5-48b1-8311-d3adb5742fd2" containerName="payload-processing"
Nov 03 18:15:56.597941 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:56.597934    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2ad7926-76d5-48b1-8311-d3adb5742fd2" containerName="payload-processing"
Nov 03 18:15:56.598105 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:56.597958    5315 cpu_manager.go:414] "containerMap: removing container" podUID="8d8504d9-e3e9-4542-8e76-3dbd404ed89f" containerName="cleanup"
Nov 03 18:15:56.598105 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:56.597963    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d8504d9-e3e9-4542-8e76-3dbd404ed89f" containerName="cleanup"
Nov 03 18:15:56.598105 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:56.597970    5315 cpu_manager.go:414] "containerMap: removing container" podUID="5d04a4ae-44e9-4f14-ada5-855c1efe7539" containerName="cleanup"
Nov 03 18:15:56.598105 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:56.597975    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d04a4ae-44e9-4f14-ada5-855c1efe7539" containerName="cleanup"
Nov 03 18:15:56.598105 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:56.597985    5315 cpu_manager.go:414] "containerMap: removing container" podUID="0f66eb4b-1a62-4dad-b70a-8fdd8f715b70" containerName="revoke-keys"
Nov 03 18:15:56.598105 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:56.597990    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f66eb4b-1a62-4dad-b70a-8fdd8f715b70" containerName="revoke-keys"
Nov 03 18:15:56.598105 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:56.598006    5315 cpu_manager.go:414] "containerMap: removing container" podUID="c06ba561-05ef-49c8-b410-6aa8cb926b93" containerName="payload-pre-processing"
Nov 03 18:15:56.598105 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:56.598011    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c06ba561-05ef-49c8-b410-6aa8cb926b93" containerName="payload-pre-processing"
Nov 03 18:15:56.598105 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:56.598088    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d04a4ae-44e9-4f14-ada5-855c1efe7539" containerName="cleanup"
Nov 03 18:15:56.598105 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:56.598100    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f66eb4b-1a62-4dad-b70a-8fdd8f715b70" containerName="revoke-keys"
Nov 03 18:15:56.598105 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:56.598107    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="c06ba561-05ef-49c8-b410-6aa8cb926b93" containerName="payload-pre-processing"
Nov 03 18:15:56.598437 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:56.598116    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="6434783b-1272-4ccd-b5c4-5a7aeeab58c7" containerName="payload-processing"
Nov 03 18:15:56.598437 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:56.598121    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="26d8c8d7-89c0-47af-b90c-8b0677f71886" containerName="cleanup"
Nov 03 18:15:56.598437 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:56.598126    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d8504d9-e3e9-4542-8e76-3dbd404ed89f" containerName="cleanup"
Nov 03 18:15:56.598437 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:56.598134    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="54bd478e-0685-4ec9-8aa0-5c3bd73c56c2" containerName="cleanup"
Nov 03 18:15:56.598437 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:56.598141    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2ad7926-76d5-48b1-8311-d3adb5742fd2" containerName="payload-processing"
Nov 03 18:15:56.600562 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:56.600539    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-80aca1-dmkj8"
Nov 03 18:15:56.695031 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:56.694994    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:15:56.695281 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:56.695030    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:15:56.695368 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:56.695035    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:15:56.695368 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:56.695103    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:15:56.698889 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:56.698863    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-80aca1-dmkj8"]
Nov 03 18:15:56.729259 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:56.729219    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdbnd\" (UniqueName: \"kubernetes.io/projected/8154fe35-e46a-4491-acb4-6990ab601e6e-kube-api-access-rdbnd\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-80aca1-dmkj8\" (UID: \"8154fe35-e46a-4491-acb4-6990ab601e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-80aca1-dmkj8"
Nov 03 18:15:56.729441 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:56.729275    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8154fe35-e46a-4491-acb4-6990ab601e6e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-80aca1-dmkj8\" (UID: \"8154fe35-e46a-4491-acb4-6990ab601e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-80aca1-dmkj8"
Nov 03 18:15:56.729441 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:56.729415    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8154fe35-e46a-4491-acb4-6990ab601e6e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-80aca1-dmkj8\" (UID: \"8154fe35-e46a-4491-acb4-6990ab601e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-80aca1-dmkj8"
Nov 03 18:15:56.830311 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:56.830275    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdbnd\" (UniqueName: \"kubernetes.io/projected/8154fe35-e46a-4491-acb4-6990ab601e6e-kube-api-access-rdbnd\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-80aca1-dmkj8\" (UID: \"8154fe35-e46a-4491-acb4-6990ab601e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-80aca1-dmkj8"
Nov 03 18:15:56.830509 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:56.830335    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8154fe35-e46a-4491-acb4-6990ab601e6e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-80aca1-dmkj8\" (UID: \"8154fe35-e46a-4491-acb4-6990ab601e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-80aca1-dmkj8"
Nov 03 18:15:56.830509 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:56.830411    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8154fe35-e46a-4491-acb4-6990ab601e6e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-80aca1-dmkj8\" (UID: \"8154fe35-e46a-4491-acb4-6990ab601e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-80aca1-dmkj8"
Nov 03 18:15:56.831089 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:56.831060    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8154fe35-e46a-4491-acb4-6990ab601e6e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-80aca1-dmkj8\" (UID: \"8154fe35-e46a-4491-acb4-6990ab601e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-80aca1-dmkj8"
Nov 03 18:15:56.832845 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:56.832822    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8154fe35-e46a-4491-acb4-6990ab601e6e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-80aca1-dmkj8\" (UID: \"8154fe35-e46a-4491-acb4-6990ab601e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-80aca1-dmkj8"
Nov 03 18:15:56.887868 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:56.887782    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdbnd\" (UniqueName: \"kubernetes.io/projected/8154fe35-e46a-4491-acb4-6990ab601e6e-kube-api-access-rdbnd\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-80aca1-dmkj8\" (UID: \"8154fe35-e46a-4491-acb4-6990ab601e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-80aca1-dmkj8"
Nov 03 18:15:56.913417 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:56.913381    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-80aca1-dmkj8"
Nov 03 18:15:57.287028 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:57.286996    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-80aca1-dmkj8"]
Nov 03 18:15:57.293561 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:15:57.293523    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8154fe35_e46a_4491_acb4_6990ab601e6e.slice/crio-d2a9c299fdd2ccd5fcebbe1aba3a1ce19fba36e5561d9a24c674cca635f5ec48 WatchSource:0}: Error finding container d2a9c299fdd2ccd5fcebbe1aba3a1ce19fba36e5561d9a24c674cca635f5ec48: Status 404 returned error can't find the container with id d2a9c299fdd2ccd5fcebbe1aba3a1ce19fba36e5561d9a24c674cca635f5ec48
Nov 03 18:15:57.764684 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:57.764642    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-80aca1-dmkj8" event={"ID":"8154fe35-e46a-4491-acb4-6990ab601e6e","Type":"ContainerStarted","Data":"101f5e7726e812113b1acb435f5bcc297b604944e7ebcd1c55de5c9e5523428c"}
Nov 03 18:15:57.765174 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:57.764708    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-80aca1-dmkj8" event={"ID":"8154fe35-e46a-4491-acb4-6990ab601e6e","Type":"ContainerStarted","Data":"d2a9c299fdd2ccd5fcebbe1aba3a1ce19fba36e5561d9a24c674cca635f5ec48"}
Nov 03 18:15:57.877375 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:57.877268    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-80aca1-dmkj8" podStartSLOduration=1.877252569 podStartE2EDuration="1.877252569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:15:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:15:57.87555949 +0000 UTC m=+1796.891757560" watchObservedRunningTime="2026-11-03 18:15:57.877252569 +0000 UTC m=+1796.893450638"
Nov 03 18:15:58.772978 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:58.772946    5315 generic.go:356] "Generic (PLEG): container finished" podID="8154fe35-e46a-4491-acb4-6990ab601e6e" containerID="101f5e7726e812113b1acb435f5bcc297b604944e7ebcd1c55de5c9e5523428c" exitCode=0
Nov 03 18:15:58.773393 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:58.773022    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-80aca1-dmkj8" event={"ID":"8154fe35-e46a-4491-acb4-6990ab601e6e","Type":"ContainerDied","Data":"101f5e7726e812113b1acb435f5bcc297b604944e7ebcd1c55de5c9e5523428c"}
Nov 03 18:15:59.949976 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:15:59.949944    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-80aca1-dmkj8"
Nov 03 18:16:00.070190 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:00.070094    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8154fe35-e46a-4491-acb4-6990ab601e6e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8154fe35-e46a-4491-acb4-6990ab601e6e-maas-api-openshift-ca\") pod \"8154fe35-e46a-4491-acb4-6990ab601e6e\" (UID: \"8154fe35-e46a-4491-acb4-6990ab601e6e\") "
Nov 03 18:16:00.070190 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:00.070140    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8154fe35-e46a-4491-acb4-6990ab601e6e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8154fe35-e46a-4491-acb4-6990ab601e6e-maas-api-xks-ca\") pod \"8154fe35-e46a-4491-acb4-6990ab601e6e\" (UID: \"8154fe35-e46a-4491-acb4-6990ab601e6e\") "
Nov 03 18:16:00.070437 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:00.070269    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8154fe35-e46a-4491-acb4-6990ab601e6e-kube-api-access-rdbnd\" (UniqueName: \"kubernetes.io/projected/8154fe35-e46a-4491-acb4-6990ab601e6e-kube-api-access-rdbnd\") pod \"8154fe35-e46a-4491-acb4-6990ab601e6e\" (UID: \"8154fe35-e46a-4491-acb4-6990ab601e6e\") "
Nov 03 18:16:00.070561 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:00.070529    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8154fe35-e46a-4491-acb4-6990ab601e6e-maas-api-openshift-ca" pod "8154fe35-e46a-4491-acb4-6990ab601e6e" (UID: "8154fe35-e46a-4491-acb4-6990ab601e6e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:16:00.070807 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:00.070780    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8154fe35-e46a-4491-acb4-6990ab601e6e-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:00.072438 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:00.072409    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8154fe35-e46a-4491-acb4-6990ab601e6e-maas-api-xks-ca" pod "8154fe35-e46a-4491-acb4-6990ab601e6e" (UID: "8154fe35-e46a-4491-acb4-6990ab601e6e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:16:00.072535 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:00.072460    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8154fe35-e46a-4491-acb4-6990ab601e6e-kube-api-access-rdbnd" pod "8154fe35-e46a-4491-acb4-6990ab601e6e" (UID: "8154fe35-e46a-4491-acb4-6990ab601e6e"). InnerVolumeSpecName "kube-api-access-rdbnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:16:00.173187 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:00.173145    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdbnd\" (UniqueName: \"kubernetes.io/projected/8154fe35-e46a-4491-acb4-6990ab601e6e-kube-api-access-rdbnd\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:00.173512 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:00.173467    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8154fe35-e46a-4491-acb4-6990ab601e6e-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:00.789722 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:00.789690    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-80aca1-dmkj8"
Nov 03 18:16:00.789909 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:00.789695    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-80aca1-dmkj8" event={"ID":"8154fe35-e46a-4491-acb4-6990ab601e6e","Type":"ContainerDied","Data":"d2a9c299fdd2ccd5fcebbe1aba3a1ce19fba36e5561d9a24c674cca635f5ec48"}
Nov 03 18:16:00.789909 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:00.789819    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2a9c299fdd2ccd5fcebbe1aba3a1ce19fba36e5561d9a24c674cca635f5ec48"
Nov 03 18:16:01.090440 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:01.090348    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74"]
Nov 03 18:16:01.091704 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:01.091661    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74" podUID="4eeee197-126d-4005-9745-e7aa8d36c574" containerName="istio-proxy" containerID="cri-o://176213a03e8120e0e463d61c18a8c649820b7e5370ddcfb8bc38714f237f1d34" gracePeriod=30
Nov 03 18:16:01.113588 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:01.113556    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479"]
Nov 03 18:16:01.113795 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:01.113774    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479" podUID="a23b3a10-d0ee-4eee-802e-97040ba5bd9d" containerName="istio-proxy" containerID="cri-o://75bf9b7fbbdfe8b707815163673abbbac96dacde2fdec6cbb3e8ad4c227fac5c" gracePeriod=30
Nov 03 18:16:01.584777 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:01.584747    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-97p65_c29d200a-c917-46e5-9cf5-21f4d952e470/ovn-acl-logging/0.log"
Nov 03 18:16:01.636234 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:01.636208    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-97p65_c29d200a-c917-46e5-9cf5-21f4d952e470/ovn-acl-logging/0.log"
Nov 03 18:16:02.977937 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:02.977900    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d"]
Nov 03 18:16:02.978347 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:02.978193    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d" podUID="a91acd67-7bae-4d10-a5b0-4f1d4d81253f" containerName="istio-proxy" containerID="cri-o://82acbf6e071bfe45aedb1bd5e5941c1ef9889be7b7a1eba99f7849266106b367" gracePeriod=30
Nov 03 18:16:03.827171 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:03.827131    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c1b2d2-2wvwd"]
Nov 03 18:16:03.827685 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:03.827668    5315 cpu_manager.go:414] "containerMap: removing container" podUID="8154fe35-e46a-4491-acb4-6990ab601e6e" containerName="revoke-keys"
Nov 03 18:16:03.827753 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:03.827687    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8154fe35-e46a-4491-acb4-6990ab601e6e" containerName="revoke-keys"
Nov 03 18:16:03.827806 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:03.827797    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="8154fe35-e46a-4491-acb4-6990ab601e6e" containerName="revoke-keys"
Nov 03 18:16:03.830529 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:03.830507    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c1b2d2-2wvwd"
Nov 03 18:16:03.872108 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:03.872069    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0a9539e-a2e7-4b78-a6c5-2697b0412c50-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c1b2d2-2wvwd\" (UID: \"d0a9539e-a2e7-4b78-a6c5-2697b0412c50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c1b2d2-2wvwd"
Nov 03 18:16:03.872108 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:03.872109    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0a9539e-a2e7-4b78-a6c5-2697b0412c50-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c1b2d2-2wvwd\" (UID: \"d0a9539e-a2e7-4b78-a6c5-2697b0412c50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c1b2d2-2wvwd"
Nov 03 18:16:03.872350 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:03.872252    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tbqj\" (UniqueName: \"kubernetes.io/projected/d0a9539e-a2e7-4b78-a6c5-2697b0412c50-kube-api-access-5tbqj\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c1b2d2-2wvwd\" (UID: \"d0a9539e-a2e7-4b78-a6c5-2697b0412c50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c1b2d2-2wvwd"
Nov 03 18:16:03.879034 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:03.879008    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:16:03.879194 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:03.879146    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:16:03.924882 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:03.924839    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:16:03.924882 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:03.924873    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:16:03.927836 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:03.927811    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c1b2d2-2wvwd"]
Nov 03 18:16:03.947932 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:03.947893    5315 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:16:03.948060 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:03.947947    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74" podUID="4eeee197-126d-4005-9745-e7aa8d36c574" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:16:03.973853 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:03.973817    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0a9539e-a2e7-4b78-a6c5-2697b0412c50-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c1b2d2-2wvwd\" (UID: \"d0a9539e-a2e7-4b78-a6c5-2697b0412c50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c1b2d2-2wvwd"
Nov 03 18:16:03.973985 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:03.973867    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0a9539e-a2e7-4b78-a6c5-2697b0412c50-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c1b2d2-2wvwd\" (UID: \"d0a9539e-a2e7-4b78-a6c5-2697b0412c50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c1b2d2-2wvwd"
Nov 03 18:16:03.973985 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:03.973940    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tbqj\" (UniqueName: \"kubernetes.io/projected/d0a9539e-a2e7-4b78-a6c5-2697b0412c50-kube-api-access-5tbqj\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c1b2d2-2wvwd\" (UID: \"d0a9539e-a2e7-4b78-a6c5-2697b0412c50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c1b2d2-2wvwd"
Nov 03 18:16:03.974647 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:03.974605    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0a9539e-a2e7-4b78-a6c5-2697b0412c50-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c1b2d2-2wvwd\" (UID: \"d0a9539e-a2e7-4b78-a6c5-2697b0412c50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c1b2d2-2wvwd"
Nov 03 18:16:03.976169 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:03.976143    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0a9539e-a2e7-4b78-a6c5-2697b0412c50-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c1b2d2-2wvwd\" (UID: \"d0a9539e-a2e7-4b78-a6c5-2697b0412c50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c1b2d2-2wvwd"
Nov 03 18:16:04.031124 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:04.031087    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tbqj\" (UniqueName: \"kubernetes.io/projected/d0a9539e-a2e7-4b78-a6c5-2697b0412c50-kube-api-access-5tbqj\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c1b2d2-2wvwd\" (UID: \"d0a9539e-a2e7-4b78-a6c5-2697b0412c50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c1b2d2-2wvwd"
Nov 03 18:16:04.145927 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:04.145837    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c1b2d2-2wvwd"
Nov 03 18:16:04.517827 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:04.517741    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c1b2d2-2wvwd"]
Nov 03 18:16:04.523341 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:16:04.523307    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0a9539e_a2e7_4b78_a6c5_2697b0412c50.slice/crio-c801fcda0b8f1a90e35141220e44500cf099f65807f8318cb60eb58c46451883 WatchSource:0}: Error finding container c801fcda0b8f1a90e35141220e44500cf099f65807f8318cb60eb58c46451883: Status 404 returned error can't find the container with id c801fcda0b8f1a90e35141220e44500cf099f65807f8318cb60eb58c46451883
Nov 03 18:16:04.735567 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:04.735538    5315 scope.go:122] "RemoveContainer" containerID="9caf8f40e07da83e465cf90633fc665f1f40aa9c9c9b9436ff819cd2a0b6de62"
Nov 03 18:16:04.751470 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:04.751441    5315 scope.go:122] "RemoveContainer" containerID="96be76605e544234a34e3708facc81660dd82760f6c083366e7e318c12a3c55a"
Nov 03 18:16:04.819497 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:04.819468    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c1b2d2-2wvwd" event={"ID":"d0a9539e-a2e7-4b78-a6c5-2697b0412c50","Type":"ContainerStarted","Data":"c801fcda0b8f1a90e35141220e44500cf099f65807f8318cb60eb58c46451883"}
Nov 03 18:16:05.838073 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:05.838036    5315 generic.go:356] "Generic (PLEG): container finished" podID="d0a9539e-a2e7-4b78-a6c5-2697b0412c50" containerID="314892e74754d6f2047867c109e23d6b10a000ed8782e15d545c2d9ab4996e53" exitCode=0
Nov 03 18:16:05.838480 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:05.838121    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c1b2d2-2wvwd" event={"ID":"d0a9539e-a2e7-4b78-a6c5-2697b0412c50","Type":"ContainerDied","Data":"314892e74754d6f2047867c109e23d6b10a000ed8782e15d545c2d9ab4996e53"}
Nov 03 18:16:06.379123 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.379092    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74"
Nov 03 18:16:06.425502 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.425474    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479"
Nov 03 18:16:06.507947 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.507911    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4eeee197-126d-4005-9745-e7aa8d36c574-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eeee197-126d-4005-9745-e7aa8d36c574-istio-podinfo\") pod \"4eeee197-126d-4005-9745-e7aa8d36c574\" (UID: \"4eeee197-126d-4005-9745-e7aa8d36c574\") "
Nov 03 18:16:06.508108 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.507970    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eeee197-126d-4005-9745-e7aa8d36c574-kube-api-access-ghrhb\" (UniqueName: \"kubernetes.io/projected/4eeee197-126d-4005-9745-e7aa8d36c574-kube-api-access-ghrhb\") pod \"4eeee197-126d-4005-9745-e7aa8d36c574\" (UID: \"4eeee197-126d-4005-9745-e7aa8d36c574\") "
Nov 03 18:16:06.508232 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.508213    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eeee197-126d-4005-9745-e7aa8d36c574-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eeee197-126d-4005-9745-e7aa8d36c574-istio-envoy\") pod \"4eeee197-126d-4005-9745-e7aa8d36c574\" (UID: \"4eeee197-126d-4005-9745-e7aa8d36c574\") "
Nov 03 18:16:06.508407 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.508335    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4eeee197-126d-4005-9745-e7aa8d36c574-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eeee197-126d-4005-9745-e7aa8d36c574-istiod-ca-cert\") pod \"4eeee197-126d-4005-9745-e7aa8d36c574\" (UID: \"4eeee197-126d-4005-9745-e7aa8d36c574\") "
Nov 03 18:16:06.509183 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.508541    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-kube-api-access-jncnn\" (UniqueName: \"kubernetes.io/projected/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-kube-api-access-jncnn\") pod \"a23b3a10-d0ee-4eee-802e-97040ba5bd9d\" (UID: \"a23b3a10-d0ee-4eee-802e-97040ba5bd9d\") "
Nov 03 18:16:06.509183 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.508579    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eeee197-126d-4005-9745-e7aa8d36c574-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eeee197-126d-4005-9745-e7aa8d36c574-credential-socket\") pod \"4eeee197-126d-4005-9745-e7aa8d36c574\" (UID: \"4eeee197-126d-4005-9745-e7aa8d36c574\") "
Nov 03 18:16:06.509183 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.508747    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eeee197-126d-4005-9745-e7aa8d36c574-istiod-ca-cert" pod "4eeee197-126d-4005-9745-e7aa8d36c574" (UID: "4eeee197-126d-4005-9745-e7aa8d36c574"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:16:06.509183 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.508849    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eeee197-126d-4005-9745-e7aa8d36c574-credential-socket" pod "4eeee197-126d-4005-9745-e7aa8d36c574" (UID: "4eeee197-126d-4005-9745-e7aa8d36c574"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:06.509540 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.509511    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-istio-token\" (UniqueName: \"kubernetes.io/projected/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-istio-token\") pod \"a23b3a10-d0ee-4eee-802e-97040ba5bd9d\" (UID: \"a23b3a10-d0ee-4eee-802e-97040ba5bd9d\") "
Nov 03 18:16:06.509696 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.509593    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eeee197-126d-4005-9745-e7aa8d36c574-istio-token\" (UniqueName: \"kubernetes.io/projected/4eeee197-126d-4005-9745-e7aa8d36c574-istio-token\") pod \"4eeee197-126d-4005-9745-e7aa8d36c574\" (UID: \"4eeee197-126d-4005-9745-e7aa8d36c574\") "
Nov 03 18:16:06.509696 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.509653    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eeee197-126d-4005-9745-e7aa8d36c574-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eeee197-126d-4005-9745-e7aa8d36c574-workload-certs\") pod \"4eeee197-126d-4005-9745-e7aa8d36c574\" (UID: \"4eeee197-126d-4005-9745-e7aa8d36c574\") "
Nov 03 18:16:06.509829 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.509718    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-istio-podinfo\") pod \"a23b3a10-d0ee-4eee-802e-97040ba5bd9d\" (UID: \"a23b3a10-d0ee-4eee-802e-97040ba5bd9d\") "
Nov 03 18:16:06.509829 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.509804    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-workload-certs\") pod \"a23b3a10-d0ee-4eee-802e-97040ba5bd9d\" (UID: \"a23b3a10-d0ee-4eee-802e-97040ba5bd9d\") "
Nov 03 18:16:06.509941 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.509854    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-istiod-ca-cert\") pod \"a23b3a10-d0ee-4eee-802e-97040ba5bd9d\" (UID: \"a23b3a10-d0ee-4eee-802e-97040ba5bd9d\") "
Nov 03 18:16:06.509941 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.509899    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-workload-socket\") pod \"a23b3a10-d0ee-4eee-802e-97040ba5bd9d\" (UID: \"a23b3a10-d0ee-4eee-802e-97040ba5bd9d\") "
Nov 03 18:16:06.511200 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.509943    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-istio-data\") pod \"a23b3a10-d0ee-4eee-802e-97040ba5bd9d\" (UID: \"a23b3a10-d0ee-4eee-802e-97040ba5bd9d\") "
Nov 03 18:16:06.511200 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.510023    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eeee197-126d-4005-9745-e7aa8d36c574-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eeee197-126d-4005-9745-e7aa8d36c574-workload-socket\") pod \"4eeee197-126d-4005-9745-e7aa8d36c574\" (UID: \"4eeee197-126d-4005-9745-e7aa8d36c574\") "
Nov 03 18:16:06.511200 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.510061    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-credential-socket\") pod \"a23b3a10-d0ee-4eee-802e-97040ba5bd9d\" (UID: \"a23b3a10-d0ee-4eee-802e-97040ba5bd9d\") "
Nov 03 18:16:06.511200 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.510101    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eeee197-126d-4005-9745-e7aa8d36c574-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eeee197-126d-4005-9745-e7aa8d36c574-istio-data\") pod \"4eeee197-126d-4005-9745-e7aa8d36c574\" (UID: \"4eeee197-126d-4005-9745-e7aa8d36c574\") "
Nov 03 18:16:06.511200 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.510149    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-istio-envoy\") pod \"a23b3a10-d0ee-4eee-802e-97040ba5bd9d\" (UID: \"a23b3a10-d0ee-4eee-802e-97040ba5bd9d\") "
Nov 03 18:16:06.511200 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.510334    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eeee197-126d-4005-9745-e7aa8d36c574-workload-certs" pod "4eeee197-126d-4005-9745-e7aa8d36c574" (UID: "4eeee197-126d-4005-9745-e7aa8d36c574"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:06.511200 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.510855    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eeee197-126d-4005-9745-e7aa8d36c574-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:06.511200 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.510883    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eeee197-126d-4005-9745-e7aa8d36c574-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:06.511200 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.510884    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eeee197-126d-4005-9745-e7aa8d36c574-istio-envoy" pod "4eeee197-126d-4005-9745-e7aa8d36c574" (UID: "4eeee197-126d-4005-9745-e7aa8d36c574"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:06.511200 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.510900    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eeee197-126d-4005-9745-e7aa8d36c574-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:06.511200 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.510972    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4eeee197-126d-4005-9745-e7aa8d36c574-istio-podinfo" pod "4eeee197-126d-4005-9745-e7aa8d36c574" (UID: "4eeee197-126d-4005-9745-e7aa8d36c574"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:16:06.511200 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.510989    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eeee197-126d-4005-9745-e7aa8d36c574-kube-api-access-ghrhb" pod "4eeee197-126d-4005-9745-e7aa8d36c574" (UID: "4eeee197-126d-4005-9745-e7aa8d36c574"). InnerVolumeSpecName "kube-api-access-ghrhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:16:06.511200 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.511137    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-workload-certs" pod "a23b3a10-d0ee-4eee-802e-97040ba5bd9d" (UID: "a23b3a10-d0ee-4eee-802e-97040ba5bd9d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:06.511907 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.511280    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-istio-data" pod "a23b3a10-d0ee-4eee-802e-97040ba5bd9d" (UID: "a23b3a10-d0ee-4eee-802e-97040ba5bd9d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:06.511907 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.511393    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-workload-socket" pod "a23b3a10-d0ee-4eee-802e-97040ba5bd9d" (UID: "a23b3a10-d0ee-4eee-802e-97040ba5bd9d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:06.511907 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.511416    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-istiod-ca-cert" pod "a23b3a10-d0ee-4eee-802e-97040ba5bd9d" (UID: "a23b3a10-d0ee-4eee-802e-97040ba5bd9d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:16:06.511907 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.511505    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-credential-socket" pod "a23b3a10-d0ee-4eee-802e-97040ba5bd9d" (UID: "a23b3a10-d0ee-4eee-802e-97040ba5bd9d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:06.511907 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.511625    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eeee197-126d-4005-9745-e7aa8d36c574-istio-data" pod "4eeee197-126d-4005-9745-e7aa8d36c574" (UID: "4eeee197-126d-4005-9745-e7aa8d36c574"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:06.511907 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.511668    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eeee197-126d-4005-9745-e7aa8d36c574-workload-socket" pod "4eeee197-126d-4005-9745-e7aa8d36c574" (UID: "4eeee197-126d-4005-9745-e7aa8d36c574"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:06.512440 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.512415    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-kube-api-access-jncnn" pod "a23b3a10-d0ee-4eee-802e-97040ba5bd9d" (UID: "a23b3a10-d0ee-4eee-802e-97040ba5bd9d"). InnerVolumeSpecName "kube-api-access-jncnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:16:06.512669 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.512643    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-istio-token" pod "a23b3a10-d0ee-4eee-802e-97040ba5bd9d" (UID: "a23b3a10-d0ee-4eee-802e-97040ba5bd9d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:16:06.513333 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.513305    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eeee197-126d-4005-9745-e7aa8d36c574-istio-token" pod "4eeee197-126d-4005-9745-e7aa8d36c574" (UID: "4eeee197-126d-4005-9745-e7aa8d36c574"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:16:06.513412 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.513334    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-istio-envoy" pod "a23b3a10-d0ee-4eee-802e-97040ba5bd9d" (UID: "a23b3a10-d0ee-4eee-802e-97040ba5bd9d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:06.513795 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.513779    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-istio-podinfo" pod "a23b3a10-d0ee-4eee-802e-97040ba5bd9d" (UID: "a23b3a10-d0ee-4eee-802e-97040ba5bd9d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:16:06.612283 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.612196    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jncnn\" (UniqueName: \"kubernetes.io/projected/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-kube-api-access-jncnn\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:06.612283 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.612232    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:06.612283 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.612245    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4eeee197-126d-4005-9745-e7aa8d36c574-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:06.612283 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.612258    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:06.612283 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.612271    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:06.612283 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.612283    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:06.612676 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.612295    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:06.612676 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.612307    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:06.612676 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.612321    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eeee197-126d-4005-9745-e7aa8d36c574-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:06.612676 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.612334    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:06.612676 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.612347    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eeee197-126d-4005-9745-e7aa8d36c574-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:06.612676 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.612358    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a23b3a10-d0ee-4eee-802e-97040ba5bd9d-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:06.612676 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.612371    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eeee197-126d-4005-9745-e7aa8d36c574-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:06.612676 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.612384    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghrhb\" (UniqueName: \"kubernetes.io/projected/4eeee197-126d-4005-9745-e7aa8d36c574-kube-api-access-ghrhb\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:06.612676 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.612397    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eeee197-126d-4005-9745-e7aa8d36c574-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:06.847531 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.847491    5315 generic.go:356] "Generic (PLEG): container finished" podID="a23b3a10-d0ee-4eee-802e-97040ba5bd9d" containerID="75bf9b7fbbdfe8b707815163673abbbac96dacde2fdec6cbb3e8ad4c227fac5c" exitCode=0
Nov 03 18:16:06.848068 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.847548    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479"
Nov 03 18:16:06.848068 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.847574    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479" event={"ID":"a23b3a10-d0ee-4eee-802e-97040ba5bd9d","Type":"ContainerDied","Data":"75bf9b7fbbdfe8b707815163673abbbac96dacde2fdec6cbb3e8ad4c227fac5c"}
Nov 03 18:16:06.848068 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.847676    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479" event={"ID":"a23b3a10-d0ee-4eee-802e-97040ba5bd9d","Type":"ContainerDied","Data":"a252e91cc245d28f00a21fd7d6dce60a4e15c505944c88ab2f1b9c93c151f918"}
Nov 03 18:16:06.848068 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.847714    5315 scope.go:122] "RemoveContainer" containerID="75bf9b7fbbdfe8b707815163673abbbac96dacde2fdec6cbb3e8ad4c227fac5c"
Nov 03 18:16:06.849269 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.849242    5315 generic.go:356] "Generic (PLEG): container finished" podID="4eeee197-126d-4005-9745-e7aa8d36c574" containerID="176213a03e8120e0e463d61c18a8c649820b7e5370ddcfb8bc38714f237f1d34" exitCode=0
Nov 03 18:16:06.850762 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.849310    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74"
Nov 03 18:16:06.850762 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.849325    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74" event={"ID":"4eeee197-126d-4005-9745-e7aa8d36c574","Type":"ContainerDied","Data":"176213a03e8120e0e463d61c18a8c649820b7e5370ddcfb8bc38714f237f1d34"}
Nov 03 18:16:06.850762 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.849468    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74" event={"ID":"4eeee197-126d-4005-9745-e7aa8d36c574","Type":"ContainerDied","Data":"4ba33d75a9c7fb05c4bf5229412b42c0fba20ffab0e02e3651de4bf486fe0c56"}
Nov 03 18:16:06.862394 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.862326    5315 scope.go:122] "RemoveContainer" containerID="75bf9b7fbbdfe8b707815163673abbbac96dacde2fdec6cbb3e8ad4c227fac5c"
Nov 03 18:16:06.862761 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:16:06.862735    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75bf9b7fbbdfe8b707815163673abbbac96dacde2fdec6cbb3e8ad4c227fac5c\": container with ID starting with 75bf9b7fbbdfe8b707815163673abbbac96dacde2fdec6cbb3e8ad4c227fac5c not found: ID does not exist" containerID="75bf9b7fbbdfe8b707815163673abbbac96dacde2fdec6cbb3e8ad4c227fac5c"
Nov 03 18:16:06.862880 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.862770    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75bf9b7fbbdfe8b707815163673abbbac96dacde2fdec6cbb3e8ad4c227fac5c"} err="failed to get container status \"75bf9b7fbbdfe8b707815163673abbbac96dacde2fdec6cbb3e8ad4c227fac5c\": rpc error: code = NotFound desc = could not find container \"75bf9b7fbbdfe8b707815163673abbbac96dacde2fdec6cbb3e8ad4c227fac5c\": container with ID starting with 75bf9b7fbbdfe8b707815163673abbbac96dacde2fdec6cbb3e8ad4c227fac5c not found: ID does not exist"
Nov 03 18:16:06.862880 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.862789    5315 scope.go:122] "RemoveContainer" containerID="176213a03e8120e0e463d61c18a8c649820b7e5370ddcfb8bc38714f237f1d34"
Nov 03 18:16:06.873072 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.873048    5315 scope.go:122] "RemoveContainer" containerID="176213a03e8120e0e463d61c18a8c649820b7e5370ddcfb8bc38714f237f1d34"
Nov 03 18:16:06.873413 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:16:06.873393    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"176213a03e8120e0e463d61c18a8c649820b7e5370ddcfb8bc38714f237f1d34\": container with ID starting with 176213a03e8120e0e463d61c18a8c649820b7e5370ddcfb8bc38714f237f1d34 not found: ID does not exist" containerID="176213a03e8120e0e463d61c18a8c649820b7e5370ddcfb8bc38714f237f1d34"
Nov 03 18:16:06.873483 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.873428    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"176213a03e8120e0e463d61c18a8c649820b7e5370ddcfb8bc38714f237f1d34"} err="failed to get container status \"176213a03e8120e0e463d61c18a8c649820b7e5370ddcfb8bc38714f237f1d34\": rpc error: code = NotFound desc = could not find container \"176213a03e8120e0e463d61c18a8c649820b7e5370ddcfb8bc38714f237f1d34\": container with ID starting with 176213a03e8120e0e463d61c18a8c649820b7e5370ddcfb8bc38714f237f1d34 not found: ID does not exist"
Nov 03 18:16:06.992411 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:06.992384    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c1b2d2-2wvwd"
Nov 03 18:16:07.002488 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:07.002452    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479"]
Nov 03 18:16:07.048583 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:07.048537    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-2q479"]
Nov 03 18:16:07.104388 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:07.104356    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a23b3a10-d0ee-4eee-802e-97040ba5bd9d" path="/var/lib/kubelet/pods/a23b3a10-d0ee-4eee-802e-97040ba5bd9d/volumes"
Nov 03 18:16:07.108442 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:07.108412    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74"]
Nov 03 18:16:07.112900 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:07.112835    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-5zw74"]
Nov 03 18:16:07.120917 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:07.120895    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0a9539e-a2e7-4b78-a6c5-2697b0412c50-kube-api-access-5tbqj\" (UniqueName: \"kubernetes.io/projected/d0a9539e-a2e7-4b78-a6c5-2697b0412c50-kube-api-access-5tbqj\") pod \"d0a9539e-a2e7-4b78-a6c5-2697b0412c50\" (UID: \"d0a9539e-a2e7-4b78-a6c5-2697b0412c50\") "
Nov 03 18:16:07.121035 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:07.120941    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0a9539e-a2e7-4b78-a6c5-2697b0412c50-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0a9539e-a2e7-4b78-a6c5-2697b0412c50-maas-api-xks-ca\") pod \"d0a9539e-a2e7-4b78-a6c5-2697b0412c50\" (UID: \"d0a9539e-a2e7-4b78-a6c5-2697b0412c50\") "
Nov 03 18:16:07.121035 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:07.120977    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0a9539e-a2e7-4b78-a6c5-2697b0412c50-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0a9539e-a2e7-4b78-a6c5-2697b0412c50-maas-api-openshift-ca\") pod \"d0a9539e-a2e7-4b78-a6c5-2697b0412c50\" (UID: \"d0a9539e-a2e7-4b78-a6c5-2697b0412c50\") "
Nov 03 18:16:07.121440 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:07.121409    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0a9539e-a2e7-4b78-a6c5-2697b0412c50-maas-api-openshift-ca" pod "d0a9539e-a2e7-4b78-a6c5-2697b0412c50" (UID: "d0a9539e-a2e7-4b78-a6c5-2697b0412c50"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:16:07.123104 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:07.123081    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0a9539e-a2e7-4b78-a6c5-2697b0412c50-kube-api-access-5tbqj" pod "d0a9539e-a2e7-4b78-a6c5-2697b0412c50" (UID: "d0a9539e-a2e7-4b78-a6c5-2697b0412c50"). InnerVolumeSpecName "kube-api-access-5tbqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:16:07.123197 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:07.123124    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0a9539e-a2e7-4b78-a6c5-2697b0412c50-maas-api-xks-ca" pod "d0a9539e-a2e7-4b78-a6c5-2697b0412c50" (UID: "d0a9539e-a2e7-4b78-a6c5-2697b0412c50"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:16:07.222849 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:07.222814    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0a9539e-a2e7-4b78-a6c5-2697b0412c50-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:07.222849 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:07.222842    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0a9539e-a2e7-4b78-a6c5-2697b0412c50-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:07.222849 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:07.222852    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tbqj\" (UniqueName: \"kubernetes.io/projected/d0a9539e-a2e7-4b78-a6c5-2697b0412c50-kube-api-access-5tbqj\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:07.860413 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:07.860384    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c1b2d2-2wvwd"
Nov 03 18:16:07.860892 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:07.860419    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c1b2d2-2wvwd" event={"ID":"d0a9539e-a2e7-4b78-a6c5-2697b0412c50","Type":"ContainerDied","Data":"c801fcda0b8f1a90e35141220e44500cf099f65807f8318cb60eb58c46451883"}
Nov 03 18:16:07.860892 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:07.860472    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c801fcda0b8f1a90e35141220e44500cf099f65807f8318cb60eb58c46451883"
Nov 03 18:16:08.247917 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:08.247889    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d"
Nov 03 18:16:08.337319 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:08.337281    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-istiod-ca-cert\") pod \"a91acd67-7bae-4d10-a5b0-4f1d4d81253f\" (UID: \"a91acd67-7bae-4d10-a5b0-4f1d4d81253f\") "
Nov 03 18:16:08.337512 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:08.337346    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-istio-podinfo\") pod \"a91acd67-7bae-4d10-a5b0-4f1d4d81253f\" (UID: \"a91acd67-7bae-4d10-a5b0-4f1d4d81253f\") "
Nov 03 18:16:08.337512 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:08.337448    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-istio-data\") pod \"a91acd67-7bae-4d10-a5b0-4f1d4d81253f\" (UID: \"a91acd67-7bae-4d10-a5b0-4f1d4d81253f\") "
Nov 03 18:16:08.337692 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:08.337516    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-istio-envoy\") pod \"a91acd67-7bae-4d10-a5b0-4f1d4d81253f\" (UID: \"a91acd67-7bae-4d10-a5b0-4f1d4d81253f\") "
Nov 03 18:16:08.337692 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:08.337559    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-kube-api-access-kvv58\" (UniqueName: \"kubernetes.io/projected/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-kube-api-access-kvv58\") pod \"a91acd67-7bae-4d10-a5b0-4f1d4d81253f\" (UID: \"a91acd67-7bae-4d10-a5b0-4f1d4d81253f\") "
Nov 03 18:16:08.337692 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:08.337599    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-workload-socket\") pod \"a91acd67-7bae-4d10-a5b0-4f1d4d81253f\" (UID: \"a91acd67-7bae-4d10-a5b0-4f1d4d81253f\") "
Nov 03 18:16:08.337692 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:08.337649    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-istio-token\" (UniqueName: \"kubernetes.io/projected/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-istio-token\") pod \"a91acd67-7bae-4d10-a5b0-4f1d4d81253f\" (UID: \"a91acd67-7bae-4d10-a5b0-4f1d4d81253f\") "
Nov 03 18:16:08.337906 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:08.337712    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-istiod-ca-cert" pod "a91acd67-7bae-4d10-a5b0-4f1d4d81253f" (UID: "a91acd67-7bae-4d10-a5b0-4f1d4d81253f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:16:08.337906 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:08.337722    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-credential-socket\") pod \"a91acd67-7bae-4d10-a5b0-4f1d4d81253f\" (UID: \"a91acd67-7bae-4d10-a5b0-4f1d4d81253f\") "
Nov 03 18:16:08.337906 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:08.337810    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-workload-certs\") pod \"a91acd67-7bae-4d10-a5b0-4f1d4d81253f\" (UID: \"a91acd67-7bae-4d10-a5b0-4f1d4d81253f\") "
Nov 03 18:16:08.338060 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:08.337947    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-credential-socket" pod "a91acd67-7bae-4d10-a5b0-4f1d4d81253f" (UID: "a91acd67-7bae-4d10-a5b0-4f1d4d81253f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:08.338377 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:08.338139    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-workload-socket" pod "a91acd67-7bae-4d10-a5b0-4f1d4d81253f" (UID: "a91acd67-7bae-4d10-a5b0-4f1d4d81253f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:08.338750 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:08.338484    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-workload-certs" pod "a91acd67-7bae-4d10-a5b0-4f1d4d81253f" (UID: "a91acd67-7bae-4d10-a5b0-4f1d4d81253f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:08.338750 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:08.338503    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:08.338750 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:08.338522    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:08.338750 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:08.338537    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:08.338750 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:08.338113    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-istio-data" pod "a91acd67-7bae-4d10-a5b0-4f1d4d81253f" (UID: "a91acd67-7bae-4d10-a5b0-4f1d4d81253f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:08.340145 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:08.340123    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-istio-podinfo" pod "a91acd67-7bae-4d10-a5b0-4f1d4d81253f" (UID: "a91acd67-7bae-4d10-a5b0-4f1d4d81253f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:16:08.340657 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:08.340638    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-kube-api-access-kvv58" pod "a91acd67-7bae-4d10-a5b0-4f1d4d81253f" (UID: "a91acd67-7bae-4d10-a5b0-4f1d4d81253f"). InnerVolumeSpecName "kube-api-access-kvv58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:16:08.340895 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:08.340880    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-istio-envoy" pod "a91acd67-7bae-4d10-a5b0-4f1d4d81253f" (UID: "a91acd67-7bae-4d10-a5b0-4f1d4d81253f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:08.341018 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:08.341004    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-istio-token" pod "a91acd67-7bae-4d10-a5b0-4f1d4d81253f" (UID: "a91acd67-7bae-4d10-a5b0-4f1d4d81253f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:16:08.439979 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:08.439888    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:08.439979 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:08.439918    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:08.439979 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:08.439931    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvv58\" (UniqueName: \"kubernetes.io/projected/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-kube-api-access-kvv58\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:08.439979 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:08.439944    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:08.439979 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:08.439956    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:08.439979 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:08.439968    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a91acd67-7bae-4d10-a5b0-4f1d4d81253f-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:08.869880 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:08.869839    5315 generic.go:356] "Generic (PLEG): container finished" podID="a91acd67-7bae-4d10-a5b0-4f1d4d81253f" containerID="82acbf6e071bfe45aedb1bd5e5941c1ef9889be7b7a1eba99f7849266106b367" exitCode=0
Nov 03 18:16:08.870370 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:08.869914    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d"
Nov 03 18:16:08.870370 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:08.869931    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d" event={"ID":"a91acd67-7bae-4d10-a5b0-4f1d4d81253f","Type":"ContainerDied","Data":"82acbf6e071bfe45aedb1bd5e5941c1ef9889be7b7a1eba99f7849266106b367"}
Nov 03 18:16:08.870370 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:08.870021    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d" event={"ID":"a91acd67-7bae-4d10-a5b0-4f1d4d81253f","Type":"ContainerDied","Data":"e59092dc0fc453af64e3e1f625b3939226187ebc7cda16b952f17df1a7a6253f"}
Nov 03 18:16:08.870370 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:08.870048    5315 scope.go:122] "RemoveContainer" containerID="82acbf6e071bfe45aedb1bd5e5941c1ef9889be7b7a1eba99f7849266106b367"
Nov 03 18:16:08.880418 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:08.880400    5315 scope.go:122] "RemoveContainer" containerID="5a807df74fdb3586c786c96521db6b16a152470bd59c19a4be212fcd36135eaa"
Nov 03 18:16:08.890053 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:08.890030    5315 scope.go:122] "RemoveContainer" containerID="82acbf6e071bfe45aedb1bd5e5941c1ef9889be7b7a1eba99f7849266106b367"
Nov 03 18:16:08.890357 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:16:08.890337    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82acbf6e071bfe45aedb1bd5e5941c1ef9889be7b7a1eba99f7849266106b367\": container with ID starting with 82acbf6e071bfe45aedb1bd5e5941c1ef9889be7b7a1eba99f7849266106b367 not found: ID does not exist" containerID="82acbf6e071bfe45aedb1bd5e5941c1ef9889be7b7a1eba99f7849266106b367"
Nov 03 18:16:08.890436 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:08.890370    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82acbf6e071bfe45aedb1bd5e5941c1ef9889be7b7a1eba99f7849266106b367"} err="failed to get container status \"82acbf6e071bfe45aedb1bd5e5941c1ef9889be7b7a1eba99f7849266106b367\": rpc error: code = NotFound desc = could not find container \"82acbf6e071bfe45aedb1bd5e5941c1ef9889be7b7a1eba99f7849266106b367\": container with ID starting with 82acbf6e071bfe45aedb1bd5e5941c1ef9889be7b7a1eba99f7849266106b367 not found: ID does not exist"
Nov 03 18:16:08.890436 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:08.890393    5315 scope.go:122] "RemoveContainer" containerID="5a807df74fdb3586c786c96521db6b16a152470bd59c19a4be212fcd36135eaa"
Nov 03 18:16:08.890687 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:16:08.890660    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a807df74fdb3586c786c96521db6b16a152470bd59c19a4be212fcd36135eaa\": container with ID starting with 5a807df74fdb3586c786c96521db6b16a152470bd59c19a4be212fcd36135eaa not found: ID does not exist" containerID="5a807df74fdb3586c786c96521db6b16a152470bd59c19a4be212fcd36135eaa"
Nov 03 18:16:08.890739 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:08.890699    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a807df74fdb3586c786c96521db6b16a152470bd59c19a4be212fcd36135eaa"} err="failed to get container status \"5a807df74fdb3586c786c96521db6b16a152470bd59c19a4be212fcd36135eaa\": rpc error: code = NotFound desc = could not find container \"5a807df74fdb3586c786c96521db6b16a152470bd59c19a4be212fcd36135eaa\": container with ID starting with 5a807df74fdb3586c786c96521db6b16a152470bd59c19a4be212fcd36135eaa not found: ID does not exist"
Nov 03 18:16:09.024478 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:09.024440    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d"]
Nov 03 18:16:09.070640 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:09.070559    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7a7c76-openshift-default-c544c5c69-r862d"]
Nov 03 18:16:09.103966 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:09.103930    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eeee197-126d-4005-9745-e7aa8d36c574" path="/var/lib/kubelet/pods/4eeee197-126d-4005-9745-e7aa8d36c574/volumes"
Nov 03 18:16:09.104426 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:09.104412    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a91acd67-7bae-4d10-a5b0-4f1d4d81253f" path="/var/lib/kubelet/pods/a91acd67-7bae-4d10-a5b0-4f1d4d81253f/volumes"
Nov 03 18:16:11.277630 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:11.276678    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1720c00c-xm2ll"]
Nov 03 18:16:11.320116 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:11.320075    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1720c00c-xm2ll"]
Nov 03 18:16:13.106914 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:13.106544    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22a6a997-394b-4c8b-9de9-171c4d22ced1" path="/var/lib/kubelet/pods/22a6a997-394b-4c8b-9de9-171c4d22ced1/volumes"
Nov 03 18:16:15.303883 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:15.303848    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c1b2d2-7c79954dcf-7htl8"]
Nov 03 18:16:15.413410 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:15.413378    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c1b2d2-6557dc56cf-kc2p2"]
Nov 03 18:16:20.305138 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:20.305092    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c1b2d2-7c79954dcf-7htl8" podUID="3801739a-23aa-4cb1-9de5-b1aa0851fa93" containerName="payload-processing" containerID="cri-o://145f57e9ab112665d43e09123ffbf9aed0b8be18f9ffeb5bcc02010b79fc6a78" gracePeriod=25
Nov 03 18:16:20.413955 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:20.413888    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c1b2d2-6557dc56cf-kc2p2" podUID="d12d9244-eac0-4c66-b63d-14147ef13e3e" containerName="payload-pre-processing" containerID="cri-o://b56d3e3f7162e8af080dfca48d0cc4fa85eedcd629229632e3a2971fd101b13e" gracePeriod=25
Nov 03 18:16:20.571650 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:20.571569    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c1b2d2-7c79954dcf-7htl8"
Nov 03 18:16:20.640229 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:20.640188    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3801739a-23aa-4cb1-9de5-b1aa0851fa93-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3801739a-23aa-4cb1-9de5-b1aa0851fa93-plugins-config-volume\") pod \"3801739a-23aa-4cb1-9de5-b1aa0851fa93\" (UID: \"3801739a-23aa-4cb1-9de5-b1aa0851fa93\") "
Nov 03 18:16:20.640402 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:20.640361    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3801739a-23aa-4cb1-9de5-b1aa0851fa93-kube-api-access-zl59j\" (UniqueName: \"kubernetes.io/projected/3801739a-23aa-4cb1-9de5-b1aa0851fa93-kube-api-access-zl59j\") pod \"3801739a-23aa-4cb1-9de5-b1aa0851fa93\" (UID: \"3801739a-23aa-4cb1-9de5-b1aa0851fa93\") "
Nov 03 18:16:20.640638 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:20.640588    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3801739a-23aa-4cb1-9de5-b1aa0851fa93-plugins-config-volume" pod "3801739a-23aa-4cb1-9de5-b1aa0851fa93" (UID: "3801739a-23aa-4cb1-9de5-b1aa0851fa93"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:16:20.640884 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:20.640864    5315 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3801739a-23aa-4cb1-9de5-b1aa0851fa93-plugins-config-volume\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:20.642935 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:20.642905    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3801739a-23aa-4cb1-9de5-b1aa0851fa93-kube-api-access-zl59j" pod "3801739a-23aa-4cb1-9de5-b1aa0851fa93" (UID: "3801739a-23aa-4cb1-9de5-b1aa0851fa93"). InnerVolumeSpecName "kube-api-access-zl59j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:16:20.685343 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:20.685316    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c1b2d2-6557dc56cf-kc2p2"
Nov 03 18:16:20.741923 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:20.741888    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d12d9244-eac0-4c66-b63d-14147ef13e3e-kube-api-access-s69lv\" (UniqueName: \"kubernetes.io/projected/d12d9244-eac0-4c66-b63d-14147ef13e3e-kube-api-access-s69lv\") pod \"d12d9244-eac0-4c66-b63d-14147ef13e3e\" (UID: \"d12d9244-eac0-4c66-b63d-14147ef13e3e\") "
Nov 03 18:16:20.742109 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:20.741939    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d12d9244-eac0-4c66-b63d-14147ef13e3e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d12d9244-eac0-4c66-b63d-14147ef13e3e-plugins-config-volume\") pod \"d12d9244-eac0-4c66-b63d-14147ef13e3e\" (UID: \"d12d9244-eac0-4c66-b63d-14147ef13e3e\") "
Nov 03 18:16:20.742370 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:20.742341    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d12d9244-eac0-4c66-b63d-14147ef13e3e-plugins-config-volume" pod "d12d9244-eac0-4c66-b63d-14147ef13e3e" (UID: "d12d9244-eac0-4c66-b63d-14147ef13e3e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:16:20.742415 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:20.742372    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl59j\" (UniqueName: \"kubernetes.io/projected/3801739a-23aa-4cb1-9de5-b1aa0851fa93-kube-api-access-zl59j\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:20.744038 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:20.744017    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d12d9244-eac0-4c66-b63d-14147ef13e3e-kube-api-access-s69lv" pod "d12d9244-eac0-4c66-b63d-14147ef13e3e" (UID: "d12d9244-eac0-4c66-b63d-14147ef13e3e"). InnerVolumeSpecName "kube-api-access-s69lv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:16:20.843673 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:20.843558    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s69lv\" (UniqueName: \"kubernetes.io/projected/d12d9244-eac0-4c66-b63d-14147ef13e3e-kube-api-access-s69lv\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:20.843673 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:20.843595    5315 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d12d9244-eac0-4c66-b63d-14147ef13e3e-plugins-config-volume\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:21.015303 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:21.015266    5315 generic.go:356] "Generic (PLEG): container finished" podID="3801739a-23aa-4cb1-9de5-b1aa0851fa93" containerID="145f57e9ab112665d43e09123ffbf9aed0b8be18f9ffeb5bcc02010b79fc6a78" exitCode=0
Nov 03 18:16:21.015496 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:21.015320    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c1b2d2-7c79954dcf-7htl8"
Nov 03 18:16:21.015496 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:21.015369    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c1b2d2-7c79954dcf-7htl8" event={"ID":"3801739a-23aa-4cb1-9de5-b1aa0851fa93","Type":"ContainerDied","Data":"145f57e9ab112665d43e09123ffbf9aed0b8be18f9ffeb5bcc02010b79fc6a78"}
Nov 03 18:16:21.015496 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:21.015431    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c1b2d2-7c79954dcf-7htl8" event={"ID":"3801739a-23aa-4cb1-9de5-b1aa0851fa93","Type":"ContainerDied","Data":"0626fbf37c0368a47db28849fe9535c5fa5fc411feeef51ce53e98178bb4056d"}
Nov 03 18:16:21.015496 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:21.015461    5315 scope.go:122] "RemoveContainer" containerID="145f57e9ab112665d43e09123ffbf9aed0b8be18f9ffeb5bcc02010b79fc6a78"
Nov 03 18:16:21.017011 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:21.016985    5315 generic.go:356] "Generic (PLEG): container finished" podID="d12d9244-eac0-4c66-b63d-14147ef13e3e" containerID="b56d3e3f7162e8af080dfca48d0cc4fa85eedcd629229632e3a2971fd101b13e" exitCode=0
Nov 03 18:16:21.017133 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:21.017029    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c1b2d2-6557dc56cf-kc2p2"
Nov 03 18:16:21.017133 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:21.017062    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c1b2d2-6557dc56cf-kc2p2" event={"ID":"d12d9244-eac0-4c66-b63d-14147ef13e3e","Type":"ContainerDied","Data":"b56d3e3f7162e8af080dfca48d0cc4fa85eedcd629229632e3a2971fd101b13e"}
Nov 03 18:16:21.017133 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:21.017124    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c1b2d2-6557dc56cf-kc2p2" event={"ID":"d12d9244-eac0-4c66-b63d-14147ef13e3e","Type":"ContainerDied","Data":"02d78d8270ec880a2f2fa960c2c6ef05c8e426f092acd737c0f50fa14ceb8de1"}
Nov 03 18:16:21.026171 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:21.026151    5315 scope.go:122] "RemoveContainer" containerID="145f57e9ab112665d43e09123ffbf9aed0b8be18f9ffeb5bcc02010b79fc6a78"
Nov 03 18:16:21.026527 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:16:21.026501    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"145f57e9ab112665d43e09123ffbf9aed0b8be18f9ffeb5bcc02010b79fc6a78\": container with ID starting with 145f57e9ab112665d43e09123ffbf9aed0b8be18f9ffeb5bcc02010b79fc6a78 not found: ID does not exist" containerID="145f57e9ab112665d43e09123ffbf9aed0b8be18f9ffeb5bcc02010b79fc6a78"
Nov 03 18:16:21.026629 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:21.026539    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"145f57e9ab112665d43e09123ffbf9aed0b8be18f9ffeb5bcc02010b79fc6a78"} err="failed to get container status \"145f57e9ab112665d43e09123ffbf9aed0b8be18f9ffeb5bcc02010b79fc6a78\": rpc error: code = NotFound desc = could not find container \"145f57e9ab112665d43e09123ffbf9aed0b8be18f9ffeb5bcc02010b79fc6a78\": container with ID starting with 145f57e9ab112665d43e09123ffbf9aed0b8be18f9ffeb5bcc02010b79fc6a78 not found: ID does not exist"
Nov 03 18:16:21.026629 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:21.026562    5315 scope.go:122] "RemoveContainer" containerID="b56d3e3f7162e8af080dfca48d0cc4fa85eedcd629229632e3a2971fd101b13e"
Nov 03 18:16:21.038195 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:21.038174    5315 scope.go:122] "RemoveContainer" containerID="b56d3e3f7162e8af080dfca48d0cc4fa85eedcd629229632e3a2971fd101b13e"
Nov 03 18:16:21.038519 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:16:21.038495    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b56d3e3f7162e8af080dfca48d0cc4fa85eedcd629229632e3a2971fd101b13e\": container with ID starting with b56d3e3f7162e8af080dfca48d0cc4fa85eedcd629229632e3a2971fd101b13e not found: ID does not exist" containerID="b56d3e3f7162e8af080dfca48d0cc4fa85eedcd629229632e3a2971fd101b13e"
Nov 03 18:16:21.038584 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:21.038529    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b56d3e3f7162e8af080dfca48d0cc4fa85eedcd629229632e3a2971fd101b13e"} err="failed to get container status \"b56d3e3f7162e8af080dfca48d0cc4fa85eedcd629229632e3a2971fd101b13e\": rpc error: code = NotFound desc = could not find container \"b56d3e3f7162e8af080dfca48d0cc4fa85eedcd629229632e3a2971fd101b13e\": container with ID starting with b56d3e3f7162e8af080dfca48d0cc4fa85eedcd629229632e3a2971fd101b13e not found: ID does not exist"
Nov 03 18:16:21.168998 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:21.168910    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c1b2d2-6557dc56cf-kc2p2"]
Nov 03 18:16:21.174395 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:21.174364    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c1b2d2-6557dc56cf-kc2p2"]
Nov 03 18:16:21.274095 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:21.274062    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c1b2d2-7c79954dcf-7htl8"]
Nov 03 18:16:21.277901 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:21.277873    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c1b2d2-7c79954dcf-7htl8"]
Nov 03 18:16:22.009077 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:22.009040    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1720c00c-7f49c64bcf-dnv6v"]
Nov 03 18:16:22.009586 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:22.009568    5315 cpu_manager.go:414] "containerMap: removing container" podUID="a23b3a10-d0ee-4eee-802e-97040ba5bd9d" containerName="istio-proxy"
Nov 03 18:16:22.009690 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:22.009589    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a23b3a10-d0ee-4eee-802e-97040ba5bd9d" containerName="istio-proxy"
Nov 03 18:16:22.009690 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:22.009605    5315 cpu_manager.go:414] "containerMap: removing container" podUID="d12d9244-eac0-4c66-b63d-14147ef13e3e" containerName="payload-pre-processing"
Nov 03 18:16:22.009690 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:22.009643    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d12d9244-eac0-4c66-b63d-14147ef13e3e" containerName="payload-pre-processing"
Nov 03 18:16:22.009690 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:22.009655    5315 cpu_manager.go:414] "containerMap: removing container" podUID="d0a9539e-a2e7-4b78-a6c5-2697b0412c50" containerName="revoke-keys"
Nov 03 18:16:22.009690 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:22.009662    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0a9539e-a2e7-4b78-a6c5-2697b0412c50" containerName="revoke-keys"
Nov 03 18:16:22.009690 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:22.009678    5315 cpu_manager.go:414] "containerMap: removing container" podUID="3801739a-23aa-4cb1-9de5-b1aa0851fa93" containerName="payload-processing"
Nov 03 18:16:22.009690 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:22.009683    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3801739a-23aa-4cb1-9de5-b1aa0851fa93" containerName="payload-processing"
Nov 03 18:16:22.009963 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:22.009699    5315 cpu_manager.go:414] "containerMap: removing container" podUID="4eeee197-126d-4005-9745-e7aa8d36c574" containerName="istio-proxy"
Nov 03 18:16:22.009963 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:22.009704    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eeee197-126d-4005-9745-e7aa8d36c574" containerName="istio-proxy"
Nov 03 18:16:22.009963 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:22.009719    5315 cpu_manager.go:414] "containerMap: removing container" podUID="a91acd67-7bae-4d10-a5b0-4f1d4d81253f" containerName="istio-proxy"
Nov 03 18:16:22.009963 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:22.009724    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a91acd67-7bae-4d10-a5b0-4f1d4d81253f" containerName="istio-proxy"
Nov 03 18:16:22.009963 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:22.009821    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="a91acd67-7bae-4d10-a5b0-4f1d4d81253f" containerName="istio-proxy"
Nov 03 18:16:22.009963 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:22.009837    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="a23b3a10-d0ee-4eee-802e-97040ba5bd9d" containerName="istio-proxy"
Nov 03 18:16:22.009963 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:22.009844    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="d12d9244-eac0-4c66-b63d-14147ef13e3e" containerName="payload-pre-processing"
Nov 03 18:16:22.009963 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:22.009851    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0a9539e-a2e7-4b78-a6c5-2697b0412c50" containerName="revoke-keys"
Nov 03 18:16:22.009963 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:22.009856    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="3801739a-23aa-4cb1-9de5-b1aa0851fa93" containerName="payload-processing"
Nov 03 18:16:22.009963 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:22.009862    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eeee197-126d-4005-9745-e7aa8d36c574" containerName="istio-proxy"
Nov 03 18:16:22.043631 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:22.020382    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1720c00c-7f49c64bcf-dnv6v"
Nov 03 18:16:22.068325 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:22.058978    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skv7q\" (UniqueName: \"kubernetes.io/projected/2c20ab79-c080-49c3-a84e-f8f3fc0713f2-kube-api-access-skv7q\") pod \"payload-processing-e2e-mt-1720c00c-7f49c64bcf-dnv6v\" (UID: \"2c20ab79-c080-49c3-a84e-f8f3fc0713f2\") " pod="openshift-ingress/payload-processing-e2e-mt-1720c00c-7f49c64bcf-dnv6v"
Nov 03 18:16:22.068325 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:22.059163    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c20ab79-c080-49c3-a84e-f8f3fc0713f2-plugins-config-volume\") pod \"payload-processing-e2e-mt-1720c00c-7f49c64bcf-dnv6v\" (UID: \"2c20ab79-c080-49c3-a84e-f8f3fc0713f2\") " pod="openshift-ingress/payload-processing-e2e-mt-1720c00c-7f49c64bcf-dnv6v"
Nov 03 18:16:22.093633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:22.089743    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-1720c00c-dockercfg-tg969\""
Nov 03 18:16:22.100525 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:22.091278    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-1720c00c\""
Nov 03 18:16:22.117682 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:22.109164    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1720c00c-7f49c64bcf-dnv6v"]
Nov 03 18:16:22.159703 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:22.159662    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c20ab79-c080-49c3-a84e-f8f3fc0713f2-plugins-config-volume\") pod \"payload-processing-e2e-mt-1720c00c-7f49c64bcf-dnv6v\" (UID: \"2c20ab79-c080-49c3-a84e-f8f3fc0713f2\") " pod="openshift-ingress/payload-processing-e2e-mt-1720c00c-7f49c64bcf-dnv6v"
Nov 03 18:16:22.159916 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:22.159739    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skv7q\" (UniqueName: \"kubernetes.io/projected/2c20ab79-c080-49c3-a84e-f8f3fc0713f2-kube-api-access-skv7q\") pod \"payload-processing-e2e-mt-1720c00c-7f49c64bcf-dnv6v\" (UID: \"2c20ab79-c080-49c3-a84e-f8f3fc0713f2\") " pod="openshift-ingress/payload-processing-e2e-mt-1720c00c-7f49c64bcf-dnv6v"
Nov 03 18:16:22.160404 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:22.160381    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c20ab79-c080-49c3-a84e-f8f3fc0713f2-plugins-config-volume\") pod \"payload-processing-e2e-mt-1720c00c-7f49c64bcf-dnv6v\" (UID: \"2c20ab79-c080-49c3-a84e-f8f3fc0713f2\") " pod="openshift-ingress/payload-processing-e2e-mt-1720c00c-7f49c64bcf-dnv6v"
Nov 03 18:16:22.197473 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:22.197440    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1720c00c-549d8c448c-lzrvl"]
Nov 03 18:16:22.197970 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:22.197956    5315 cpu_manager.go:414] "containerMap: removing container" podUID="a91acd67-7bae-4d10-a5b0-4f1d4d81253f" containerName="istio-proxy"
Nov 03 18:16:22.198036 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:22.197971    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a91acd67-7bae-4d10-a5b0-4f1d4d81253f" containerName="istio-proxy"
Nov 03 18:16:22.198074 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:22.198055    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="a91acd67-7bae-4d10-a5b0-4f1d4d81253f" containerName="istio-proxy"
Nov 03 18:16:22.200870 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:22.200850    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1720c00c-549d8c448c-lzrvl"
Nov 03 18:16:22.216839 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:22.216815    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skv7q\" (UniqueName: \"kubernetes.io/projected/2c20ab79-c080-49c3-a84e-f8f3fc0713f2-kube-api-access-skv7q\") pod \"payload-processing-e2e-mt-1720c00c-7f49c64bcf-dnv6v\" (UID: \"2c20ab79-c080-49c3-a84e-f8f3fc0713f2\") " pod="openshift-ingress/payload-processing-e2e-mt-1720c00c-7f49c64bcf-dnv6v"
Nov 03 18:16:22.261031 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:22.260962    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc4effb6-ba04-4eab-aeaf-a80ef45d2acb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1720c00c-549d8c448c-lzrvl\" (UID: \"fc4effb6-ba04-4eab-aeaf-a80ef45d2acb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1720c00c-549d8c448c-lzrvl"
Nov 03 18:16:22.261031 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:22.260996    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np78v\" (UniqueName: \"kubernetes.io/projected/fc4effb6-ba04-4eab-aeaf-a80ef45d2acb-kube-api-access-np78v\") pod \"payload-pre-processing-e2e-mt-1720c00c-549d8c448c-lzrvl\" (UID: \"fc4effb6-ba04-4eab-aeaf-a80ef45d2acb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1720c00c-549d8c448c-lzrvl"
Nov 03 18:16:22.262285 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:22.262265    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1720c00c-549d8c448c-lzrvl"]
Nov 03 18:16:22.362318 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:22.362278    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc4effb6-ba04-4eab-aeaf-a80ef45d2acb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1720c00c-549d8c448c-lzrvl\" (UID: \"fc4effb6-ba04-4eab-aeaf-a80ef45d2acb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1720c00c-549d8c448c-lzrvl"
Nov 03 18:16:22.362318 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:22.362320    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np78v\" (UniqueName: \"kubernetes.io/projected/fc4effb6-ba04-4eab-aeaf-a80ef45d2acb-kube-api-access-np78v\") pod \"payload-pre-processing-e2e-mt-1720c00c-549d8c448c-lzrvl\" (UID: \"fc4effb6-ba04-4eab-aeaf-a80ef45d2acb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1720c00c-549d8c448c-lzrvl"
Nov 03 18:16:22.362987 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:22.362967    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc4effb6-ba04-4eab-aeaf-a80ef45d2acb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1720c00c-549d8c448c-lzrvl\" (UID: \"fc4effb6-ba04-4eab-aeaf-a80ef45d2acb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1720c00c-549d8c448c-lzrvl"
Nov 03 18:16:22.419996 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:22.419963    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np78v\" (UniqueName: \"kubernetes.io/projected/fc4effb6-ba04-4eab-aeaf-a80ef45d2acb-kube-api-access-np78v\") pod \"payload-pre-processing-e2e-mt-1720c00c-549d8c448c-lzrvl\" (UID: \"fc4effb6-ba04-4eab-aeaf-a80ef45d2acb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1720c00c-549d8c448c-lzrvl"
Nov 03 18:16:22.431332 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:22.431305    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1720c00c-7f49c64bcf-dnv6v"
Nov 03 18:16:22.513868 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:22.513771    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1720c00c-549d8c448c-lzrvl"
Nov 03 18:16:22.803384 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:22.803350    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1720c00c-7f49c64bcf-dnv6v"]
Nov 03 18:16:22.809239 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:16:22.809209    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c20ab79_c080_49c3_a84e_f8f3fc0713f2.slice/crio-88a5df0fa8bbb5e3593e7cf5c514395972f532cb451f95eca4666e5fbd59fea6 WatchSource:0}: Error finding container 88a5df0fa8bbb5e3593e7cf5c514395972f532cb451f95eca4666e5fbd59fea6: Status 404 returned error can't find the container with id 88a5df0fa8bbb5e3593e7cf5c514395972f532cb451f95eca4666e5fbd59fea6
Nov 03 18:16:22.905085 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:22.905046    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1720c00c-549d8c448c-lzrvl"]
Nov 03 18:16:22.911097 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:16:22.911062    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc4effb6_ba04_4eab_aeaf_a80ef45d2acb.slice/crio-124db5dc6265f8960896e94f938160f510f5aa55de42b8020bc5d0ed0f82d405 WatchSource:0}: Error finding container 124db5dc6265f8960896e94f938160f510f5aa55de42b8020bc5d0ed0f82d405: Status 404 returned error can't find the container with id 124db5dc6265f8960896e94f938160f510f5aa55de42b8020bc5d0ed0f82d405
Nov 03 18:16:23.104170 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:23.104067    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3801739a-23aa-4cb1-9de5-b1aa0851fa93" path="/var/lib/kubelet/pods/3801739a-23aa-4cb1-9de5-b1aa0851fa93/volumes"
Nov 03 18:16:23.104731 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:23.104710    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d12d9244-eac0-4c66-b63d-14147ef13e3e" path="/var/lib/kubelet/pods/d12d9244-eac0-4c66-b63d-14147ef13e3e/volumes"
Nov 03 18:16:23.126118 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:23.126082    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1720c00c-549d8c448c-lzrvl" event={"ID":"fc4effb6-ba04-4eab-aeaf-a80ef45d2acb","Type":"ContainerStarted","Data":"2f7089861bfcff1ed9a705ed7d9c7d4f15071ddaa018022f4ea24c181ed3b10c"}
Nov 03 18:16:23.126319 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:23.126147    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1720c00c-549d8c448c-lzrvl" event={"ID":"fc4effb6-ba04-4eab-aeaf-a80ef45d2acb","Type":"ContainerStarted","Data":"124db5dc6265f8960896e94f938160f510f5aa55de42b8020bc5d0ed0f82d405"}
Nov 03 18:16:23.126319 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:23.126305    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-1720c00c-549d8c448c-lzrvl"
Nov 03 18:16:23.128031 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:23.128002    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1720c00c-7f49c64bcf-dnv6v" event={"ID":"2c20ab79-c080-49c3-a84e-f8f3fc0713f2","Type":"ContainerStarted","Data":"439eb85ffd330e0f188e970555768dd2944b1512da3f7852af9f1d779fb5408a"}
Nov 03 18:16:23.128162 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:23.128042    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1720c00c-7f49c64bcf-dnv6v" event={"ID":"2c20ab79-c080-49c3-a84e-f8f3fc0713f2","Type":"ContainerStarted","Data":"88a5df0fa8bbb5e3593e7cf5c514395972f532cb451f95eca4666e5fbd59fea6"}
Nov 03 18:16:23.128162 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:23.128098    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-1720c00c-7f49c64bcf-dnv6v"
Nov 03 18:16:23.235093 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:23.235035    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-1720c00c-549d8c448c-lzrvl" podStartSLOduration=1.235021082 podStartE2EDuration="1.235021082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:16:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:16:23.233519137 +0000 UTC m=+1822.249717206" watchObservedRunningTime="2026-11-03 18:16:23.235021082 +0000 UTC m=+1822.251219200"
Nov 03 18:16:23.360659 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:23.344004    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-1720c00c-7f49c64bcf-dnv6v" podStartSLOduration=2.343987007 podStartE2EDuration="2.343987007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:16:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:16:23.341049386 +0000 UTC m=+1822.357247455" watchObservedRunningTime="2026-11-03 18:16:23.343987007 +0000 UTC m=+1822.360185075"
Nov 03 18:16:24.577453 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:16:24.577415    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:16:34.137889 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:34.137855    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-1720c00c-7f49c64bcf-dnv6v"
Nov 03 18:16:34.138295 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:34.137931    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-1720c00c-549d8c448c-lzrvl"
Nov 03 18:16:35.059645 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:35.059583    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77185c03-2nn69"]
Nov 03 18:16:35.094338 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:35.094283    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77185c03-2nn69"]
Nov 03 18:16:35.112061 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:35.112024    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69511ea9-d925-43c9-a861-5cf16ee6b490" path="/var/lib/kubelet/pods/69511ea9-d925-43c9-a861-5cf16ee6b490/volumes"
Nov 03 18:16:36.807349 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:36.807313    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c1b2d2-2wvwd"]
Nov 03 18:16:36.853653 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:36.853620    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c1b2d2-2wvwd"]
Nov 03 18:16:37.104095 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:37.104018    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0a9539e-a2e7-4b78-a6c5-2697b0412c50" path="/var/lib/kubelet/pods/d0a9539e-a2e7-4b78-a6c5-2697b0412c50/volumes"
Nov 03 18:16:42.562468 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:42.562429    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-80aca1-dmkj8"]
Nov 03 18:16:42.608795 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:42.608763    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-80aca1-dmkj8"]
Nov 03 18:16:43.103681 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:43.103645    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8154fe35-e46a-4491-acb4-6990ab601e6e" path="/var/lib/kubelet/pods/8154fe35-e46a-4491-acb4-6990ab601e6e/volumes"
Nov 03 18:16:43.846706 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:43.846671    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56"]
Nov 03 18:16:43.847165 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:43.846909    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56" podUID="7296122e-d510-4826-9a6a-2f663d6603bf" containerName="istio-proxy" containerID="cri-o://fa2f6550978008e98297621aacbd87dab09e8452943faa71a97fb27069e9e68d" gracePeriod=30
Nov 03 18:16:44.803506 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:44.803468    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f79c02-b4bk8"]
Nov 03 18:16:44.815464 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:44.815431    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f79c02-b4bk8"
Nov 03 18:16:44.865288 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:44.865219    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:16:44.865833 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:44.865323    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:16:44.865833 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:44.865374    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:16:44.865833 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:44.865825    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:16:44.868136 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:44.868112    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f79c02-b4bk8"]
Nov 03 18:16:44.983163 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:44.983113    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c977dc59-b70d-4cc0-9531-d76c172d7b52-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-f79c02-b4bk8\" (UID: \"c977dc59-b70d-4cc0-9531-d76c172d7b52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f79c02-b4bk8"
Nov 03 18:16:44.983163 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:44.983166    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-php75\" (UniqueName: \"kubernetes.io/projected/c977dc59-b70d-4cc0-9531-d76c172d7b52-kube-api-access-php75\") pod \"maas-api-revoke-keys-e2e-worker-w2-f79c02-b4bk8\" (UID: \"c977dc59-b70d-4cc0-9531-d76c172d7b52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f79c02-b4bk8"
Nov 03 18:16:44.983389 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:44.983270    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c977dc59-b70d-4cc0-9531-d76c172d7b52-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-f79c02-b4bk8\" (UID: \"c977dc59-b70d-4cc0-9531-d76c172d7b52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f79c02-b4bk8"
Nov 03 18:16:45.085067 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:45.084972    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c977dc59-b70d-4cc0-9531-d76c172d7b52-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-f79c02-b4bk8\" (UID: \"c977dc59-b70d-4cc0-9531-d76c172d7b52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f79c02-b4bk8"
Nov 03 18:16:45.085067 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:45.085014    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-php75\" (UniqueName: \"kubernetes.io/projected/c977dc59-b70d-4cc0-9531-d76c172d7b52-kube-api-access-php75\") pod \"maas-api-revoke-keys-e2e-worker-w2-f79c02-b4bk8\" (UID: \"c977dc59-b70d-4cc0-9531-d76c172d7b52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f79c02-b4bk8"
Nov 03 18:16:45.085067 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:45.085061    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c977dc59-b70d-4cc0-9531-d76c172d7b52-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-f79c02-b4bk8\" (UID: \"c977dc59-b70d-4cc0-9531-d76c172d7b52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f79c02-b4bk8"
Nov 03 18:16:45.085694 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:45.085672    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c977dc59-b70d-4cc0-9531-d76c172d7b52-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-f79c02-b4bk8\" (UID: \"c977dc59-b70d-4cc0-9531-d76c172d7b52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f79c02-b4bk8"
Nov 03 18:16:45.087438 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:45.087416    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c977dc59-b70d-4cc0-9531-d76c172d7b52-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-f79c02-b4bk8\" (UID: \"c977dc59-b70d-4cc0-9531-d76c172d7b52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f79c02-b4bk8"
Nov 03 18:16:45.142299 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:45.142263    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-php75\" (UniqueName: \"kubernetes.io/projected/c977dc59-b70d-4cc0-9531-d76c172d7b52-kube-api-access-php75\") pod \"maas-api-revoke-keys-e2e-worker-w2-f79c02-b4bk8\" (UID: \"c977dc59-b70d-4cc0-9531-d76c172d7b52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f79c02-b4bk8"
Nov 03 18:16:45.431583 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:45.431504    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f79c02-b4bk8"
Nov 03 18:16:45.801061 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:45.801026    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f79c02-b4bk8"]
Nov 03 18:16:45.808396 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:16:45.808363    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc977dc59_b70d_4cc0_9531_d76c172d7b52.slice/crio-6b65e1bbf5e820e86ad0281d7d0484a98fd7d4ab579a2e1285e29b6142e5d7ef WatchSource:0}: Error finding container 6b65e1bbf5e820e86ad0281d7d0484a98fd7d4ab579a2e1285e29b6142e5d7ef: Status 404 returned error can't find the container with id 6b65e1bbf5e820e86ad0281d7d0484a98fd7d4ab579a2e1285e29b6142e5d7ef
Nov 03 18:16:46.301884 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:46.301843    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f79c02-b4bk8" event={"ID":"c977dc59-b70d-4cc0-9531-d76c172d7b52","Type":"ContainerStarted","Data":"8d52257469951c2068c5f5823a43128fb9db800497f04b6c1fcf43586b8a81de"}
Nov 03 18:16:46.302313 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:46.301911    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f79c02-b4bk8" event={"ID":"c977dc59-b70d-4cc0-9531-d76c172d7b52","Type":"ContainerStarted","Data":"6b65e1bbf5e820e86ad0281d7d0484a98fd7d4ab579a2e1285e29b6142e5d7ef"}
Nov 03 18:16:46.411250 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:46.411147    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f79c02-b4bk8" podStartSLOduration=2.411133596 podStartE2EDuration="2.411133596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:16:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:16:46.408733697 +0000 UTC m=+1845.424931766" watchObservedRunningTime="2026-11-03 18:16:46.411133596 +0000 UTC m=+1845.427331666"
Nov 03 18:16:47.309907 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:47.309873    5315 generic.go:356] "Generic (PLEG): container finished" podID="c977dc59-b70d-4cc0-9531-d76c172d7b52" containerID="8d52257469951c2068c5f5823a43128fb9db800497f04b6c1fcf43586b8a81de" exitCode=0
Nov 03 18:16:47.310343 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:47.309921    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f79c02-b4bk8" event={"ID":"c977dc59-b70d-4cc0-9531-d76c172d7b52","Type":"ContainerDied","Data":"8d52257469951c2068c5f5823a43128fb9db800497f04b6c1fcf43586b8a81de"}
Nov 03 18:16:48.464902 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:48.464870    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f79c02-b4bk8"
Nov 03 18:16:48.647764 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:48.643545    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c977dc59-b70d-4cc0-9531-d76c172d7b52-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c977dc59-b70d-4cc0-9531-d76c172d7b52-maas-api-openshift-ca\") pod \"c977dc59-b70d-4cc0-9531-d76c172d7b52\" (UID: \"c977dc59-b70d-4cc0-9531-d76c172d7b52\") "
Nov 03 18:16:48.658812 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:48.658773    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c977dc59-b70d-4cc0-9531-d76c172d7b52-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c977dc59-b70d-4cc0-9531-d76c172d7b52-maas-api-xks-ca\") pod \"c977dc59-b70d-4cc0-9531-d76c172d7b52\" (UID: \"c977dc59-b70d-4cc0-9531-d76c172d7b52\") "
Nov 03 18:16:48.701160 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:48.645075    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c977dc59-b70d-4cc0-9531-d76c172d7b52-maas-api-openshift-ca" pod "c977dc59-b70d-4cc0-9531-d76c172d7b52" (UID: "c977dc59-b70d-4cc0-9531-d76c172d7b52"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:16:48.701160 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:48.688831    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c977dc59-b70d-4cc0-9531-d76c172d7b52-kube-api-access-php75\" (UniqueName: \"kubernetes.io/projected/c977dc59-b70d-4cc0-9531-d76c172d7b52-kube-api-access-php75\") pod \"c977dc59-b70d-4cc0-9531-d76c172d7b52\" (UID: \"c977dc59-b70d-4cc0-9531-d76c172d7b52\") "
Nov 03 18:16:48.704358 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:48.704297    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c977dc59-b70d-4cc0-9531-d76c172d7b52-maas-api-xks-ca" pod "c977dc59-b70d-4cc0-9531-d76c172d7b52" (UID: "c977dc59-b70d-4cc0-9531-d76c172d7b52"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:16:48.708088 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:48.708051    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c977dc59-b70d-4cc0-9531-d76c172d7b52-kube-api-access-php75" pod "c977dc59-b70d-4cc0-9531-d76c172d7b52" (UID: "c977dc59-b70d-4cc0-9531-d76c172d7b52"). InnerVolumeSpecName "kube-api-access-php75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:16:48.734759 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:48.734723    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c977dc59-b70d-4cc0-9531-d76c172d7b52-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:48.735008 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:48.734988    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c977dc59-b70d-4cc0-9531-d76c172d7b52-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:48.735118 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:48.735105    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-php75\" (UniqueName: \"kubernetes.io/projected/c977dc59-b70d-4cc0-9531-d76c172d7b52-kube-api-access-php75\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:49.115812 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:49.115777    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56"
Nov 03 18:16:49.241149 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:49.241064    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7296122e-d510-4826-9a6a-2f663d6603bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7296122e-d510-4826-9a6a-2f663d6603bf-workload-certs\") pod \"7296122e-d510-4826-9a6a-2f663d6603bf\" (UID: \"7296122e-d510-4826-9a6a-2f663d6603bf\") "
Nov 03 18:16:49.241149 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:49.241109    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7296122e-d510-4826-9a6a-2f663d6603bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7296122e-d510-4826-9a6a-2f663d6603bf-istio-podinfo\") pod \"7296122e-d510-4826-9a6a-2f663d6603bf\" (UID: \"7296122e-d510-4826-9a6a-2f663d6603bf\") "
Nov 03 18:16:49.241149 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:49.241140    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7296122e-d510-4826-9a6a-2f663d6603bf-istio-token\" (UniqueName: \"kubernetes.io/projected/7296122e-d510-4826-9a6a-2f663d6603bf-istio-token\") pod \"7296122e-d510-4826-9a6a-2f663d6603bf\" (UID: \"7296122e-d510-4826-9a6a-2f663d6603bf\") "
Nov 03 18:16:49.241446 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:49.241228    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7296122e-d510-4826-9a6a-2f663d6603bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7296122e-d510-4826-9a6a-2f663d6603bf-istio-data\") pod \"7296122e-d510-4826-9a6a-2f663d6603bf\" (UID: \"7296122e-d510-4826-9a6a-2f663d6603bf\") "
Nov 03 18:16:49.241446 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:49.241327    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7296122e-d510-4826-9a6a-2f663d6603bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7296122e-d510-4826-9a6a-2f663d6603bf-workload-socket\") pod \"7296122e-d510-4826-9a6a-2f663d6603bf\" (UID: \"7296122e-d510-4826-9a6a-2f663d6603bf\") "
Nov 03 18:16:49.241446 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:49.241368    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7296122e-d510-4826-9a6a-2f663d6603bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7296122e-d510-4826-9a6a-2f663d6603bf-credential-socket\") pod \"7296122e-d510-4826-9a6a-2f663d6603bf\" (UID: \"7296122e-d510-4826-9a6a-2f663d6603bf\") "
Nov 03 18:16:49.241446 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:49.241421    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7296122e-d510-4826-9a6a-2f663d6603bf-workload-certs" pod "7296122e-d510-4826-9a6a-2f663d6603bf" (UID: "7296122e-d510-4826-9a6a-2f663d6603bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:49.241706 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:49.241432    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7296122e-d510-4826-9a6a-2f663d6603bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7296122e-d510-4826-9a6a-2f663d6603bf-istiod-ca-cert\") pod \"7296122e-d510-4826-9a6a-2f663d6603bf\" (UID: \"7296122e-d510-4826-9a6a-2f663d6603bf\") "
Nov 03 18:16:49.241706 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:49.241514    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7296122e-d510-4826-9a6a-2f663d6603bf-kube-api-access-2d6dk\" (UniqueName: \"kubernetes.io/projected/7296122e-d510-4826-9a6a-2f663d6603bf-kube-api-access-2d6dk\") pod \"7296122e-d510-4826-9a6a-2f663d6603bf\" (UID: \"7296122e-d510-4826-9a6a-2f663d6603bf\") "
Nov 03 18:16:49.241706 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:49.241562    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7296122e-d510-4826-9a6a-2f663d6603bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7296122e-d510-4826-9a6a-2f663d6603bf-istio-envoy\") pod \"7296122e-d510-4826-9a6a-2f663d6603bf\" (UID: \"7296122e-d510-4826-9a6a-2f663d6603bf\") "
Nov 03 18:16:49.241885 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:49.241837    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7296122e-d510-4826-9a6a-2f663d6603bf-istio-data" pod "7296122e-d510-4826-9a6a-2f663d6603bf" (UID: "7296122e-d510-4826-9a6a-2f663d6603bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:49.241885 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:49.241850    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7296122e-d510-4826-9a6a-2f663d6603bf-workload-socket" pod "7296122e-d510-4826-9a6a-2f663d6603bf" (UID: "7296122e-d510-4826-9a6a-2f663d6603bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:49.242275 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:49.242081    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7296122e-d510-4826-9a6a-2f663d6603bf-credential-socket" pod "7296122e-d510-4826-9a6a-2f663d6603bf" (UID: "7296122e-d510-4826-9a6a-2f663d6603bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:49.242419 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:49.242320    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7296122e-d510-4826-9a6a-2f663d6603bf-istiod-ca-cert" pod "7296122e-d510-4826-9a6a-2f663d6603bf" (UID: "7296122e-d510-4826-9a6a-2f663d6603bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:16:49.243146 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:49.243122    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7296122e-d510-4826-9a6a-2f663d6603bf-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:49.243264 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:49.243166    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7296122e-d510-4826-9a6a-2f663d6603bf-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:49.243264 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:49.243181    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7296122e-d510-4826-9a6a-2f663d6603bf-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:49.243264 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:49.243194    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7296122e-d510-4826-9a6a-2f663d6603bf-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:49.243264 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:49.243206    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7296122e-d510-4826-9a6a-2f663d6603bf-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:49.244510 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:49.244241    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7296122e-d510-4826-9a6a-2f663d6603bf-istio-podinfo" pod "7296122e-d510-4826-9a6a-2f663d6603bf" (UID: "7296122e-d510-4826-9a6a-2f663d6603bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:16:49.248468 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:49.245386    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7296122e-d510-4826-9a6a-2f663d6603bf-kube-api-access-2d6dk" pod "7296122e-d510-4826-9a6a-2f663d6603bf" (UID: "7296122e-d510-4826-9a6a-2f663d6603bf"). InnerVolumeSpecName "kube-api-access-2d6dk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:16:49.248468 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:49.245433    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7296122e-d510-4826-9a6a-2f663d6603bf-istio-envoy" pod "7296122e-d510-4826-9a6a-2f663d6603bf" (UID: "7296122e-d510-4826-9a6a-2f663d6603bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:49.248468 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:49.246547    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7296122e-d510-4826-9a6a-2f663d6603bf-istio-token" pod "7296122e-d510-4826-9a6a-2f663d6603bf" (UID: "7296122e-d510-4826-9a6a-2f663d6603bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:16:49.317522 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:49.317485    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt"]
Nov 03 18:16:49.326519 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:49.326485    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q"]
Nov 03 18:16:49.326931 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:49.326876    5315 generic.go:356] "Generic (PLEG): container finished" podID="7296122e-d510-4826-9a6a-2f663d6603bf" containerID="fa2f6550978008e98297621aacbd87dab09e8452943faa71a97fb27069e9e68d" exitCode=0
Nov 03 18:16:49.327066 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:49.326978    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56"
Nov 03 18:16:49.327155 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:49.327126    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56" event={"ID":"7296122e-d510-4826-9a6a-2f663d6603bf","Type":"ContainerDied","Data":"fa2f6550978008e98297621aacbd87dab09e8452943faa71a97fb27069e9e68d"}
Nov 03 18:16:49.327270 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:49.327248    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56" event={"ID":"7296122e-d510-4826-9a6a-2f663d6603bf","Type":"ContainerDied","Data":"7417c0bea5ea62ca6863e80e36b2c4f0dbb923faad6b84931debeaf4a16d4eb2"}
Nov 03 18:16:49.327351 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:49.327292    5315 scope.go:122] "RemoveContainer" containerID="fa2f6550978008e98297621aacbd87dab09e8452943faa71a97fb27069e9e68d"
Nov 03 18:16:49.330879 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:49.330841    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f79c02-b4bk8" event={"ID":"c977dc59-b70d-4cc0-9531-d76c172d7b52","Type":"ContainerDied","Data":"6b65e1bbf5e820e86ad0281d7d0484a98fd7d4ab579a2e1285e29b6142e5d7ef"}
Nov 03 18:16:49.331011 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:49.330905    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b65e1bbf5e820e86ad0281d7d0484a98fd7d4ab579a2e1285e29b6142e5d7ef"
Nov 03 18:16:49.331011 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:49.330977    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f79c02-b4bk8"
Nov 03 18:16:49.347578 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:49.347543    5315 scope.go:122] "RemoveContainer" containerID="4defaaf0656c36b5a0a57ca43a2b83dd12eaff89e5a3fe5d8ca68d955cf57dd1"
Nov 03 18:16:49.349515 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:49.349486    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d6dk\" (UniqueName: \"kubernetes.io/projected/7296122e-d510-4826-9a6a-2f663d6603bf-kube-api-access-2d6dk\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:49.349515 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:49.349515    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7296122e-d510-4826-9a6a-2f663d6603bf-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:49.349766 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:49.349526    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7296122e-d510-4826-9a6a-2f663d6603bf-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:49.349766 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:49.349536    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7296122e-d510-4826-9a6a-2f663d6603bf-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:49.360473 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:49.360448    5315 scope.go:122] "RemoveContainer" containerID="fa2f6550978008e98297621aacbd87dab09e8452943faa71a97fb27069e9e68d"
Nov 03 18:16:49.360888 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:16:49.360865    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa2f6550978008e98297621aacbd87dab09e8452943faa71a97fb27069e9e68d\": container with ID starting with fa2f6550978008e98297621aacbd87dab09e8452943faa71a97fb27069e9e68d not found: ID does not exist" containerID="fa2f6550978008e98297621aacbd87dab09e8452943faa71a97fb27069e9e68d"
Nov 03 18:16:49.360981 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:49.360899    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa2f6550978008e98297621aacbd87dab09e8452943faa71a97fb27069e9e68d"} err="failed to get container status \"fa2f6550978008e98297621aacbd87dab09e8452943faa71a97fb27069e9e68d\": rpc error: code = NotFound desc = could not find container \"fa2f6550978008e98297621aacbd87dab09e8452943faa71a97fb27069e9e68d\": container with ID starting with fa2f6550978008e98297621aacbd87dab09e8452943faa71a97fb27069e9e68d not found: ID does not exist"
Nov 03 18:16:49.360981 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:49.360919    5315 scope.go:122] "RemoveContainer" containerID="4defaaf0656c36b5a0a57ca43a2b83dd12eaff89e5a3fe5d8ca68d955cf57dd1"
Nov 03 18:16:49.361218 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:16:49.361199    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4defaaf0656c36b5a0a57ca43a2b83dd12eaff89e5a3fe5d8ca68d955cf57dd1\": container with ID starting with 4defaaf0656c36b5a0a57ca43a2b83dd12eaff89e5a3fe5d8ca68d955cf57dd1 not found: ID does not exist" containerID="4defaaf0656c36b5a0a57ca43a2b83dd12eaff89e5a3fe5d8ca68d955cf57dd1"
Nov 03 18:16:49.361281 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:49.361225    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4defaaf0656c36b5a0a57ca43a2b83dd12eaff89e5a3fe5d8ca68d955cf57dd1"} err="failed to get container status \"4defaaf0656c36b5a0a57ca43a2b83dd12eaff89e5a3fe5d8ca68d955cf57dd1\": rpc error: code = NotFound desc = could not find container \"4defaaf0656c36b5a0a57ca43a2b83dd12eaff89e5a3fe5d8ca68d955cf57dd1\": container with ID starting with 4defaaf0656c36b5a0a57ca43a2b83dd12eaff89e5a3fe5d8ca68d955cf57dd1 not found: ID does not exist"
Nov 03 18:16:49.530738 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:49.530705    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56"]
Nov 03 18:16:49.535352 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:49.535323    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c1b2d2-openshift-default-68bbc6f569-mqk56"]
Nov 03 18:16:49.835268 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:49.835181    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r"]
Nov 03 18:16:49.835843 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:49.835796    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r" podUID="36d4f867-6750-43de-8cdc-76f7e76fdd44" containerName="istio-proxy" containerID="cri-o://8d83ae71521c7ad0906394baa523c3950a1510da5c32acae2d4dd04872610854" gracePeriod=30
Nov 03 18:16:50.670962 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:50.670922    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-232f9e-h58gj"]
Nov 03 18:16:50.671441 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:50.671426    5315 cpu_manager.go:414] "containerMap: removing container" podUID="7296122e-d510-4826-9a6a-2f663d6603bf" containerName="istio-proxy"
Nov 03 18:16:50.671484 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:50.671442    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7296122e-d510-4826-9a6a-2f663d6603bf" containerName="istio-proxy"
Nov 03 18:16:50.671484 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:50.671470    5315 cpu_manager.go:414] "containerMap: removing container" podUID="7296122e-d510-4826-9a6a-2f663d6603bf" containerName="istio-proxy"
Nov 03 18:16:50.671484 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:50.671476    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7296122e-d510-4826-9a6a-2f663d6603bf" containerName="istio-proxy"
Nov 03 18:16:50.671586 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:50.671495    5315 cpu_manager.go:414] "containerMap: removing container" podUID="c977dc59-b70d-4cc0-9531-d76c172d7b52" containerName="revoke-keys"
Nov 03 18:16:50.671586 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:50.671500    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c977dc59-b70d-4cc0-9531-d76c172d7b52" containerName="revoke-keys"
Nov 03 18:16:50.671586 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:50.671569    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="7296122e-d510-4826-9a6a-2f663d6603bf" containerName="istio-proxy"
Nov 03 18:16:50.671586 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:50.671579    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="c977dc59-b70d-4cc0-9531-d76c172d7b52" containerName="revoke-keys"
Nov 03 18:16:50.676654 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:50.676626    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-232f9e-h58gj"
Nov 03 18:16:50.680968 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:50.680937    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c42d4a5-6add-4665-ac0f-0a9ea35c9158-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-232f9e-h58gj\" (UID: \"6c42d4a5-6add-4665-ac0f-0a9ea35c9158\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-232f9e-h58gj"
Nov 03 18:16:50.681110 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:50.680997    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5drt\" (UniqueName: \"kubernetes.io/projected/6c42d4a5-6add-4665-ac0f-0a9ea35c9158-kube-api-access-d5drt\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-232f9e-h58gj\" (UID: \"6c42d4a5-6add-4665-ac0f-0a9ea35c9158\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-232f9e-h58gj"
Nov 03 18:16:50.681163 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:50.681129    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c42d4a5-6add-4665-ac0f-0a9ea35c9158-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-232f9e-h58gj\" (UID: \"6c42d4a5-6add-4665-ac0f-0a9ea35c9158\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-232f9e-h58gj"
Nov 03 18:16:50.724813 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:50.724782    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:16:50.724987 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:50.724890    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:16:50.770814 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:50.770771    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:16:50.770814 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:50.770794    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:16:50.773484 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:50.773456    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-232f9e-h58gj"]
Nov 03 18:16:50.782305 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:50.782275    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c42d4a5-6add-4665-ac0f-0a9ea35c9158-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-232f9e-h58gj\" (UID: \"6c42d4a5-6add-4665-ac0f-0a9ea35c9158\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-232f9e-h58gj"
Nov 03 18:16:50.782827 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:50.782326    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5drt\" (UniqueName: \"kubernetes.io/projected/6c42d4a5-6add-4665-ac0f-0a9ea35c9158-kube-api-access-d5drt\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-232f9e-h58gj\" (UID: \"6c42d4a5-6add-4665-ac0f-0a9ea35c9158\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-232f9e-h58gj"
Nov 03 18:16:50.782827 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:50.782413    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c42d4a5-6add-4665-ac0f-0a9ea35c9158-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-232f9e-h58gj\" (UID: \"6c42d4a5-6add-4665-ac0f-0a9ea35c9158\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-232f9e-h58gj"
Nov 03 18:16:50.783063 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:50.783042    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c42d4a5-6add-4665-ac0f-0a9ea35c9158-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-232f9e-h58gj\" (UID: \"6c42d4a5-6add-4665-ac0f-0a9ea35c9158\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-232f9e-h58gj"
Nov 03 18:16:50.785382 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:50.785359    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c42d4a5-6add-4665-ac0f-0a9ea35c9158-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-232f9e-h58gj\" (UID: \"6c42d4a5-6add-4665-ac0f-0a9ea35c9158\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-232f9e-h58gj"
Nov 03 18:16:50.840215 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:50.840174    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5drt\" (UniqueName: \"kubernetes.io/projected/6c42d4a5-6add-4665-ac0f-0a9ea35c9158-kube-api-access-d5drt\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-232f9e-h58gj\" (UID: \"6c42d4a5-6add-4665-ac0f-0a9ea35c9158\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-232f9e-h58gj"
Nov 03 18:16:50.990315 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:50.990228    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-232f9e-h58gj"
Nov 03 18:16:51.106474 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:51.106435    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7296122e-d510-4826-9a6a-2f663d6603bf" path="/var/lib/kubelet/pods/7296122e-d510-4826-9a6a-2f663d6603bf/volumes"
Nov 03 18:16:51.361570 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:51.361527    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-232f9e-h58gj"]
Nov 03 18:16:51.368337 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:16:51.368307    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c42d4a5_6add_4665_ac0f_0a9ea35c9158.slice/crio-a286dc5e94abc48455e829be335dedef9b581047d468add6e0f046ec5f07cffe WatchSource:0}: Error finding container a286dc5e94abc48455e829be335dedef9b581047d468add6e0f046ec5f07cffe: Status 404 returned error can't find the container with id a286dc5e94abc48455e829be335dedef9b581047d468add6e0f046ec5f07cffe
Nov 03 18:16:52.354137 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:52.354098    5315 generic.go:356] "Generic (PLEG): container finished" podID="6c42d4a5-6add-4665-ac0f-0a9ea35c9158" containerID="0cd309e994a848d938ba0e26fa9cecf0fb5a0cdfc3858e0bf62a0d6d9c7ffbd7" exitCode=0
Nov 03 18:16:52.354520 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:52.354190    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-232f9e-h58gj" event={"ID":"6c42d4a5-6add-4665-ac0f-0a9ea35c9158","Type":"ContainerDied","Data":"0cd309e994a848d938ba0e26fa9cecf0fb5a0cdfc3858e0bf62a0d6d9c7ffbd7"}
Nov 03 18:16:52.354520 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:52.354279    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-232f9e-h58gj" event={"ID":"6c42d4a5-6add-4665-ac0f-0a9ea35c9158","Type":"ContainerStarted","Data":"a286dc5e94abc48455e829be335dedef9b581047d468add6e0f046ec5f07cffe"}
Nov 03 18:16:53.520815 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:53.520785    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-232f9e-h58gj"
Nov 03 18:16:53.618036 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:53.617950    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c42d4a5-6add-4665-ac0f-0a9ea35c9158-kube-api-access-d5drt\" (UniqueName: \"kubernetes.io/projected/6c42d4a5-6add-4665-ac0f-0a9ea35c9158-kube-api-access-d5drt\") pod \"6c42d4a5-6add-4665-ac0f-0a9ea35c9158\" (UID: \"6c42d4a5-6add-4665-ac0f-0a9ea35c9158\") "
Nov 03 18:16:53.618187 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:53.618108    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c42d4a5-6add-4665-ac0f-0a9ea35c9158-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c42d4a5-6add-4665-ac0f-0a9ea35c9158-maas-api-xks-ca\") pod \"6c42d4a5-6add-4665-ac0f-0a9ea35c9158\" (UID: \"6c42d4a5-6add-4665-ac0f-0a9ea35c9158\") "
Nov 03 18:16:53.618263 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:53.618196    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c42d4a5-6add-4665-ac0f-0a9ea35c9158-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c42d4a5-6add-4665-ac0f-0a9ea35c9158-maas-api-openshift-ca\") pod \"6c42d4a5-6add-4665-ac0f-0a9ea35c9158\" (UID: \"6c42d4a5-6add-4665-ac0f-0a9ea35c9158\") "
Nov 03 18:16:53.618708 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:53.618672    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c42d4a5-6add-4665-ac0f-0a9ea35c9158-maas-api-openshift-ca" pod "6c42d4a5-6add-4665-ac0f-0a9ea35c9158" (UID: "6c42d4a5-6add-4665-ac0f-0a9ea35c9158"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:16:53.620210 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:53.620183    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c42d4a5-6add-4665-ac0f-0a9ea35c9158-maas-api-xks-ca" pod "6c42d4a5-6add-4665-ac0f-0a9ea35c9158" (UID: "6c42d4a5-6add-4665-ac0f-0a9ea35c9158"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:16:53.620322 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:53.620265    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c42d4a5-6add-4665-ac0f-0a9ea35c9158-kube-api-access-d5drt" pod "6c42d4a5-6add-4665-ac0f-0a9ea35c9158" (UID: "6c42d4a5-6add-4665-ac0f-0a9ea35c9158"). InnerVolumeSpecName "kube-api-access-d5drt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:16:53.719760 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:53.719723    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5drt\" (UniqueName: \"kubernetes.io/projected/6c42d4a5-6add-4665-ac0f-0a9ea35c9158-kube-api-access-d5drt\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:53.719943 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:53.719780    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c42d4a5-6add-4665-ac0f-0a9ea35c9158-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:53.719943 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:53.719798    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c42d4a5-6add-4665-ac0f-0a9ea35c9158-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:54.367943 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:54.367915    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-232f9e-h58gj"
Nov 03 18:16:54.368118 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:54.367948    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-232f9e-h58gj" event={"ID":"6c42d4a5-6add-4665-ac0f-0a9ea35c9158","Type":"ContainerDied","Data":"a286dc5e94abc48455e829be335dedef9b581047d468add6e0f046ec5f07cffe"}
Nov 03 18:16:54.368118 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:54.368000    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a286dc5e94abc48455e829be335dedef9b581047d468add6e0f046ec5f07cffe"
Nov 03 18:16:54.577240 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:16:54.577201    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:16:55.091005 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:55.090979    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r"
Nov 03 18:16:55.135861 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:55.135777    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36d4f867-6750-43de-8cdc-76f7e76fdd44-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d4f867-6750-43de-8cdc-76f7e76fdd44-credential-socket\") pod \"36d4f867-6750-43de-8cdc-76f7e76fdd44\" (UID: \"36d4f867-6750-43de-8cdc-76f7e76fdd44\") "
Nov 03 18:16:55.135861 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:55.135819    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36d4f867-6750-43de-8cdc-76f7e76fdd44-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36d4f867-6750-43de-8cdc-76f7e76fdd44-istio-podinfo\") pod \"36d4f867-6750-43de-8cdc-76f7e76fdd44\" (UID: \"36d4f867-6750-43de-8cdc-76f7e76fdd44\") "
Nov 03 18:16:55.135861 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:55.135854    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36d4f867-6750-43de-8cdc-76f7e76fdd44-istio-token\" (UniqueName: \"kubernetes.io/projected/36d4f867-6750-43de-8cdc-76f7e76fdd44-istio-token\") pod \"36d4f867-6750-43de-8cdc-76f7e76fdd44\" (UID: \"36d4f867-6750-43de-8cdc-76f7e76fdd44\") "
Nov 03 18:16:55.136147 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:55.135890    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36d4f867-6750-43de-8cdc-76f7e76fdd44-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36d4f867-6750-43de-8cdc-76f7e76fdd44-istio-data\") pod \"36d4f867-6750-43de-8cdc-76f7e76fdd44\" (UID: \"36d4f867-6750-43de-8cdc-76f7e76fdd44\") "
Nov 03 18:16:55.136147 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:55.135941    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36d4f867-6750-43de-8cdc-76f7e76fdd44-kube-api-access-jp2x2\" (UniqueName: \"kubernetes.io/projected/36d4f867-6750-43de-8cdc-76f7e76fdd44-kube-api-access-jp2x2\") pod \"36d4f867-6750-43de-8cdc-76f7e76fdd44\" (UID: \"36d4f867-6750-43de-8cdc-76f7e76fdd44\") "
Nov 03 18:16:55.136147 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:55.135977    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36d4f867-6750-43de-8cdc-76f7e76fdd44-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36d4f867-6750-43de-8cdc-76f7e76fdd44-istio-envoy\") pod \"36d4f867-6750-43de-8cdc-76f7e76fdd44\" (UID: \"36d4f867-6750-43de-8cdc-76f7e76fdd44\") "
Nov 03 18:16:55.136147 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:55.136003    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36d4f867-6750-43de-8cdc-76f7e76fdd44-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36d4f867-6750-43de-8cdc-76f7e76fdd44-workload-certs\") pod \"36d4f867-6750-43de-8cdc-76f7e76fdd44\" (UID: \"36d4f867-6750-43de-8cdc-76f7e76fdd44\") "
Nov 03 18:16:55.136147 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:55.136090    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36d4f867-6750-43de-8cdc-76f7e76fdd44-credential-socket" pod "36d4f867-6750-43de-8cdc-76f7e76fdd44" (UID: "36d4f867-6750-43de-8cdc-76f7e76fdd44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:55.136147 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:55.136105    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36d4f867-6750-43de-8cdc-76f7e76fdd44-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36d4f867-6750-43de-8cdc-76f7e76fdd44-istiod-ca-cert\") pod \"36d4f867-6750-43de-8cdc-76f7e76fdd44\" (UID: \"36d4f867-6750-43de-8cdc-76f7e76fdd44\") "
Nov 03 18:16:55.136147 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:55.136149    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36d4f867-6750-43de-8cdc-76f7e76fdd44-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d4f867-6750-43de-8cdc-76f7e76fdd44-workload-socket\") pod \"36d4f867-6750-43de-8cdc-76f7e76fdd44\" (UID: \"36d4f867-6750-43de-8cdc-76f7e76fdd44\") "
Nov 03 18:16:55.136536 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:55.136361    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36d4f867-6750-43de-8cdc-76f7e76fdd44-workload-certs" pod "36d4f867-6750-43de-8cdc-76f7e76fdd44" (UID: "36d4f867-6750-43de-8cdc-76f7e76fdd44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:55.136723 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:55.136678    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36d4f867-6750-43de-8cdc-76f7e76fdd44-istio-data" pod "36d4f867-6750-43de-8cdc-76f7e76fdd44" (UID: "36d4f867-6750-43de-8cdc-76f7e76fdd44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:55.136723 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:55.136693    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36d4f867-6750-43de-8cdc-76f7e76fdd44-workload-socket" pod "36d4f867-6750-43de-8cdc-76f7e76fdd44" (UID: "36d4f867-6750-43de-8cdc-76f7e76fdd44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:55.136869 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:55.136849    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d4f867-6750-43de-8cdc-76f7e76fdd44-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:55.136937 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:55.136872    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36d4f867-6750-43de-8cdc-76f7e76fdd44-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:55.136937 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:55.136890    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36d4f867-6750-43de-8cdc-76f7e76fdd44-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:55.136937 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:55.136904    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d4f867-6750-43de-8cdc-76f7e76fdd44-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:55.137107 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:55.137010    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36d4f867-6750-43de-8cdc-76f7e76fdd44-istiod-ca-cert" pod "36d4f867-6750-43de-8cdc-76f7e76fdd44" (UID: "36d4f867-6750-43de-8cdc-76f7e76fdd44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:16:55.138308 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:55.138281    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36d4f867-6750-43de-8cdc-76f7e76fdd44-istio-podinfo" pod "36d4f867-6750-43de-8cdc-76f7e76fdd44" (UID: "36d4f867-6750-43de-8cdc-76f7e76fdd44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:16:55.138412 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:55.138346    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36d4f867-6750-43de-8cdc-76f7e76fdd44-istio-envoy" pod "36d4f867-6750-43de-8cdc-76f7e76fdd44" (UID: "36d4f867-6750-43de-8cdc-76f7e76fdd44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:55.138412 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:55.138394    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36d4f867-6750-43de-8cdc-76f7e76fdd44-kube-api-access-jp2x2" pod "36d4f867-6750-43de-8cdc-76f7e76fdd44" (UID: "36d4f867-6750-43de-8cdc-76f7e76fdd44"). InnerVolumeSpecName "kube-api-access-jp2x2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:16:55.138510 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:55.138427    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36d4f867-6750-43de-8cdc-76f7e76fdd44-istio-token" pod "36d4f867-6750-43de-8cdc-76f7e76fdd44" (UID: "36d4f867-6750-43de-8cdc-76f7e76fdd44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:16:55.238552 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:55.238511    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36d4f867-6750-43de-8cdc-76f7e76fdd44-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:55.238552 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:55.238545    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36d4f867-6750-43de-8cdc-76f7e76fdd44-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:55.238552 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:55.238554    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp2x2\" (UniqueName: \"kubernetes.io/projected/36d4f867-6750-43de-8cdc-76f7e76fdd44-kube-api-access-jp2x2\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:55.238847 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:55.238565    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36d4f867-6750-43de-8cdc-76f7e76fdd44-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:55.238847 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:55.238574    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36d4f867-6750-43de-8cdc-76f7e76fdd44-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:55.376657 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:55.376595    5315 generic.go:356] "Generic (PLEG): container finished" podID="36d4f867-6750-43de-8cdc-76f7e76fdd44" containerID="8d83ae71521c7ad0906394baa523c3950a1510da5c32acae2d4dd04872610854" exitCode=0
Nov 03 18:16:55.376657 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:55.376657    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r"
Nov 03 18:16:55.376893 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:55.376675    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r" event={"ID":"36d4f867-6750-43de-8cdc-76f7e76fdd44","Type":"ContainerDied","Data":"8d83ae71521c7ad0906394baa523c3950a1510da5c32acae2d4dd04872610854"}
Nov 03 18:16:55.376893 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:55.376731    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r" event={"ID":"36d4f867-6750-43de-8cdc-76f7e76fdd44","Type":"ContainerDied","Data":"abcbad0ff0b07a0244721069ed8e4cbad72b81f038183f3a9f6fb50ae17fad5b"}
Nov 03 18:16:55.376893 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:55.376755    5315 scope.go:122] "RemoveContainer" containerID="8d83ae71521c7ad0906394baa523c3950a1510da5c32acae2d4dd04872610854"
Nov 03 18:16:55.388355 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:55.388336    5315 scope.go:122] "RemoveContainer" containerID="16228370c5964cfa3a4a65ddaca6c939b84eebd4df14ca5f5aa8eb65090781f3"
Nov 03 18:16:55.397455 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:55.397433    5315 scope.go:122] "RemoveContainer" containerID="8d83ae71521c7ad0906394baa523c3950a1510da5c32acae2d4dd04872610854"
Nov 03 18:16:55.397754 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:16:55.397730    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d83ae71521c7ad0906394baa523c3950a1510da5c32acae2d4dd04872610854\": container with ID starting with 8d83ae71521c7ad0906394baa523c3950a1510da5c32acae2d4dd04872610854 not found: ID does not exist" containerID="8d83ae71521c7ad0906394baa523c3950a1510da5c32acae2d4dd04872610854"
Nov 03 18:16:55.397802 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:55.397762    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d83ae71521c7ad0906394baa523c3950a1510da5c32acae2d4dd04872610854"} err="failed to get container status \"8d83ae71521c7ad0906394baa523c3950a1510da5c32acae2d4dd04872610854\": rpc error: code = NotFound desc = could not find container \"8d83ae71521c7ad0906394baa523c3950a1510da5c32acae2d4dd04872610854\": container with ID starting with 8d83ae71521c7ad0906394baa523c3950a1510da5c32acae2d4dd04872610854 not found: ID does not exist"
Nov 03 18:16:55.397802 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:55.397786    5315 scope.go:122] "RemoveContainer" containerID="16228370c5964cfa3a4a65ddaca6c939b84eebd4df14ca5f5aa8eb65090781f3"
Nov 03 18:16:55.398063 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:16:55.398042    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16228370c5964cfa3a4a65ddaca6c939b84eebd4df14ca5f5aa8eb65090781f3\": container with ID starting with 16228370c5964cfa3a4a65ddaca6c939b84eebd4df14ca5f5aa8eb65090781f3 not found: ID does not exist" containerID="16228370c5964cfa3a4a65ddaca6c939b84eebd4df14ca5f5aa8eb65090781f3"
Nov 03 18:16:55.398120 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:55.398069    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16228370c5964cfa3a4a65ddaca6c939b84eebd4df14ca5f5aa8eb65090781f3"} err="failed to get container status \"16228370c5964cfa3a4a65ddaca6c939b84eebd4df14ca5f5aa8eb65090781f3\": rpc error: code = NotFound desc = could not find container \"16228370c5964cfa3a4a65ddaca6c939b84eebd4df14ca5f5aa8eb65090781f3\": container with ID starting with 16228370c5964cfa3a4a65ddaca6c939b84eebd4df14ca5f5aa8eb65090781f3 not found: ID does not exist"
Nov 03 18:16:55.532419 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:55.532373    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r"]
Nov 03 18:16:55.534842 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:55.534809    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-80aca1-openshift-default-854b4cdc85-ffm8r"]
Nov 03 18:16:56.340141 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:56.340100    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-f79c02-5c7b8dcbf7-qtwl6"]
Nov 03 18:16:56.458308 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:56.458268    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-f79c02-5477cd8fff-2h4nk"]
Nov 03 18:16:57.104265 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:16:57.104227    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36d4f867-6750-43de-8cdc-76f7e76fdd44" path="/var/lib/kubelet/pods/36d4f867-6750-43de-8cdc-76f7e76fdd44/volumes"
Nov 03 18:17:01.341123 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:01.341048    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-f79c02-5c7b8dcbf7-qtwl6" podUID="7861e9eb-a526-442b-800f-6009fa55789a" containerName="payload-processing" containerID="cri-o://cb5b039bde5463d14872c292f2434ff1fc45bd3258d6b823d2919b57416e6042" gracePeriod=25
Nov 03 18:17:01.458999 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:01.458911    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f79c02-5477cd8fff-2h4nk" podUID="13fcfa33-03ce-4384-a431-edf17a1f2735" containerName="payload-pre-processing" containerID="cri-o://da8194e643108ce1d44254e047b5d88c793a05b242660947f1578a1fd2af28ba" gracePeriod=25
Nov 03 18:17:02.161777 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:02.161737    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-232f9e-75b58576ff-2z4jb"]
Nov 03 18:17:02.269389 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:02.269335    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-232f9e-649f6fd75-thwmg"]
Nov 03 18:17:04.344532 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:04.344492    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt" podUID="7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625" containerName="main" containerID="cri-o://6b00778efce404a2ba5f3e060b5f1ecab59600daaf00de64fcc29c3c6136a552" gracePeriod=45
Nov 03 18:17:04.352189 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:04.352148    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q" podUID="c974b0f1-e00b-4e4a-b193-df769068983d" containerName="main" containerID="cri-o://e7574d3f726d08af068b3943873d9a4c520307ba566a7ac1acf20e5296e866a0" gracePeriod=45
Nov 03 18:17:04.644859 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:04.644828    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt"
Nov 03 18:17:04.648059 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:04.648039    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q"
Nov 03 18:17:04.683776 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:04.683746    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625-dshm\") pod \"7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625\" (UID: \"7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625\") "
Nov 03 18:17:04.683951 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:04.683797    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c974b0f1-e00b-4e4a-b193-df769068983d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c974b0f1-e00b-4e4a-b193-df769068983d-model-cache\") pod \"c974b0f1-e00b-4e4a-b193-df769068983d\" (UID: \"c974b0f1-e00b-4e4a-b193-df769068983d\") "
Nov 03 18:17:04.683951 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:04.683831    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625-tls-certs\" (UniqueName: \"kubernetes.io/secret/7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625-tls-certs\") pod \"7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625\" (UID: \"7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625\") "
Nov 03 18:17:04.683951 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:04.683853    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625-tmp-dir\") pod \"7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625\" (UID: \"7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625\") "
Nov 03 18:17:04.683951 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:04.683877    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c974b0f1-e00b-4e4a-b193-df769068983d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c974b0f1-e00b-4e4a-b193-df769068983d-dshm\") pod \"c974b0f1-e00b-4e4a-b193-df769068983d\" (UID: \"c974b0f1-e00b-4e4a-b193-df769068983d\") "
Nov 03 18:17:04.683951 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:04.683898    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c974b0f1-e00b-4e4a-b193-df769068983d-home\" (UniqueName: \"kubernetes.io/empty-dir/c974b0f1-e00b-4e4a-b193-df769068983d-home\") pod \"c974b0f1-e00b-4e4a-b193-df769068983d\" (UID: \"c974b0f1-e00b-4e4a-b193-df769068983d\") "
Nov 03 18:17:04.683951 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:04.683947    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c974b0f1-e00b-4e4a-b193-df769068983d-tls-certs\" (UniqueName: \"kubernetes.io/secret/c974b0f1-e00b-4e4a-b193-df769068983d-tls-certs\") pod \"c974b0f1-e00b-4e4a-b193-df769068983d\" (UID: \"c974b0f1-e00b-4e4a-b193-df769068983d\") "
Nov 03 18:17:04.684284 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:04.683983    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625-model-cache\") pod \"7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625\" (UID: \"7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625\") "
Nov 03 18:17:04.684284 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:04.684060    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625-kube-api-access-wz8tb\" (UniqueName: \"kubernetes.io/projected/7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625-kube-api-access-wz8tb\") pod \"7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625\" (UID: \"7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625\") "
Nov 03 18:17:04.684284 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:04.684157    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c974b0f1-e00b-4e4a-b193-df769068983d-model-cache" pod "c974b0f1-e00b-4e4a-b193-df769068983d" (UID: "c974b0f1-e00b-4e4a-b193-df769068983d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:04.684284 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:04.684165    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625-tmp-dir" pod "7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625" (UID: "7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:04.684284 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:04.684177    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c974b0f1-e00b-4e4a-b193-df769068983d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c974b0f1-e00b-4e4a-b193-df769068983d-tmp-dir\") pod \"c974b0f1-e00b-4e4a-b193-df769068983d\" (UID: \"c974b0f1-e00b-4e4a-b193-df769068983d\") "
Nov 03 18:17:04.684284 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:04.684266    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c974b0f1-e00b-4e4a-b193-df769068983d-kube-api-access-v2gnr\" (UniqueName: \"kubernetes.io/projected/c974b0f1-e00b-4e4a-b193-df769068983d-kube-api-access-v2gnr\") pod \"c974b0f1-e00b-4e4a-b193-df769068983d\" (UID: \"c974b0f1-e00b-4e4a-b193-df769068983d\") "
Nov 03 18:17:04.684567 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:04.684367    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c974b0f1-e00b-4e4a-b193-df769068983d-tmp-dir" pod "c974b0f1-e00b-4e4a-b193-df769068983d" (UID: "c974b0f1-e00b-4e4a-b193-df769068983d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:04.684567 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:04.684371    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625-model-cache" pod "7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625" (UID: "7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:04.684567 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:04.684381    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625-home\" (UniqueName: \"kubernetes.io/empty-dir/7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625-home\") pod \"7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625\" (UID: \"7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625\") "
Nov 03 18:17:04.684567 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:04.684510    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c974b0f1-e00b-4e4a-b193-df769068983d-home" pod "c974b0f1-e00b-4e4a-b193-df769068983d" (UID: "c974b0f1-e00b-4e4a-b193-df769068983d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:04.684934 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:04.684872    5315 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c974b0f1-e00b-4e4a-b193-df769068983d-tmp-dir\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:04.684934 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:04.684896    5315 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c974b0f1-e00b-4e4a-b193-df769068983d-model-cache\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:04.684934 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:04.684911    5315 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625-tmp-dir\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:04.684934 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:04.684924    5315 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c974b0f1-e00b-4e4a-b193-df769068983d-home\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:04.685162 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:04.684936    5315 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625-model-cache\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:04.685162 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:04.684970    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625-home" pod "7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625" (UID: "7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:04.686784 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:04.686756    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625-dshm" pod "7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625" (UID: "7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:04.686935 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:04.686812    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c974b0f1-e00b-4e4a-b193-df769068983d-dshm" pod "c974b0f1-e00b-4e4a-b193-df769068983d" (UID: "c974b0f1-e00b-4e4a-b193-df769068983d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:04.686935 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:04.686823    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625-tls-certs" pod "7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625" (UID: "7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:17:04.687052 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:04.686968    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625-kube-api-access-wz8tb" pod "7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625" (UID: "7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625"). InnerVolumeSpecName "kube-api-access-wz8tb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:17:04.687052 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:04.686991    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c974b0f1-e00b-4e4a-b193-df769068983d-kube-api-access-v2gnr" pod "c974b0f1-e00b-4e4a-b193-df769068983d" (UID: "c974b0f1-e00b-4e4a-b193-df769068983d"). InnerVolumeSpecName "kube-api-access-v2gnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:17:04.687121 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:04.687087    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c974b0f1-e00b-4e4a-b193-df769068983d-tls-certs" pod "c974b0f1-e00b-4e4a-b193-df769068983d" (UID: "c974b0f1-e00b-4e4a-b193-df769068983d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:17:04.786536 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:04.786496    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2gnr\" (UniqueName: \"kubernetes.io/projected/c974b0f1-e00b-4e4a-b193-df769068983d-kube-api-access-v2gnr\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:04.786536 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:04.786531    5315 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625-home\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:04.786536 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:04.786540    5315 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625-dshm\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:04.786536 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:04.786548    5315 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625-tls-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:04.786842 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:04.786557    5315 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c974b0f1-e00b-4e4a-b193-df769068983d-dshm\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:04.786842 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:04.786565    5315 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c974b0f1-e00b-4e4a-b193-df769068983d-tls-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:04.786842 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:04.786574    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz8tb\" (UniqueName: \"kubernetes.io/projected/7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625-kube-api-access-wz8tb\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:04.873133 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:04.873049    5315 scope.go:122] "RemoveContainer" containerID="e7574d3f726d08af068b3943873d9a4c520307ba566a7ac1acf20e5296e866a0"
Nov 03 18:17:04.886052 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:04.886019    5315 scope.go:122] "RemoveContainer" containerID="6b00778efce404a2ba5f3e060b5f1ecab59600daaf00de64fcc29c3c6136a552"
Nov 03 18:17:05.441833 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:05.441782    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt" event={"ID":"7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625","Type":"ContainerDied","Data":"6b00778efce404a2ba5f3e060b5f1ecab59600daaf00de64fcc29c3c6136a552"}
Nov 03 18:17:05.442250 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:05.441833    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt"
Nov 03 18:17:05.442250 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:05.441849    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt" event={"ID":"7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625","Type":"ContainerDied","Data":"356b4b70449b256b9f31b1dbb392c9483d9c1cc3ac777f6cc7babd37233c6b86"}
Nov 03 18:17:05.442250 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:05.441863    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q" event={"ID":"c974b0f1-e00b-4e4a-b193-df769068983d","Type":"ContainerDied","Data":"e7574d3f726d08af068b3943873d9a4c520307ba566a7ac1acf20e5296e866a0"}
Nov 03 18:17:05.442250 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:05.441877    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q" event={"ID":"c974b0f1-e00b-4e4a-b193-df769068983d","Type":"ContainerDied","Data":"31320146e8a2b9b695ed96ab2a1677fb6f1563408177578ddf2627b777af786f"}
Nov 03 18:17:05.442250 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:05.441803    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q"
Nov 03 18:17:05.761849 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:05.761767    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q"]
Nov 03 18:17:05.766037 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:05.765998    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-f79c02/premium-w2-f79c02-kserve-5875b5f8b7-snz7q"]
Nov 03 18:17:05.867085 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:05.867051    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt"]
Nov 03 18:17:05.871221 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:05.871188    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-f79c02/distinct-w2-f79c02-kserve-6c4b58696b-wxnwt"]
Nov 03 18:17:07.104515 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:07.104477    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625" path="/var/lib/kubelet/pods/7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625/volumes"
Nov 03 18:17:07.104986 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:07.104969    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c974b0f1-e00b-4e4a-b193-df769068983d" path="/var/lib/kubelet/pods/c974b0f1-e00b-4e4a-b193-df769068983d/volumes"
Nov 03 18:17:07.162104 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:07.162061    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-232f9e-75b58576ff-2z4jb" podUID="0a25b9cd-06ba-4a5f-b73f-e63177585ff2" containerName="payload-processing" containerID="cri-o://67b2b09bdc9091fac78291aadbe5b9786299a10f208a965b93cbbeb07130fb3e" gracePeriod=25
Nov 03 18:17:07.271014 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:07.270970    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-232f9e-649f6fd75-thwmg" podUID="c6ca8679-f0c2-48a0-8786-ed4cbad5295d" containerName="payload-pre-processing" containerID="cri-o://8366f8c386c20c9a31fd816b8f379cabd66ae38b01939c41eb7ea15da0cfaab6" gracePeriod=25
Nov 03 18:17:07.447710 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:07.447597    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-232f9e-75b58576ff-2z4jb"
Nov 03 18:17:07.467642 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:07.458365    5315 generic.go:356] "Generic (PLEG): container finished" podID="0a25b9cd-06ba-4a5f-b73f-e63177585ff2" containerID="67b2b09bdc9091fac78291aadbe5b9786299a10f208a965b93cbbeb07130fb3e" exitCode=0
Nov 03 18:17:07.467642 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:07.458426    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-232f9e-75b58576ff-2z4jb" event={"ID":"0a25b9cd-06ba-4a5f-b73f-e63177585ff2","Type":"ContainerDied","Data":"67b2b09bdc9091fac78291aadbe5b9786299a10f208a965b93cbbeb07130fb3e"}
Nov 03 18:17:07.467642 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:07.458489    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-232f9e-75b58576ff-2z4jb" event={"ID":"0a25b9cd-06ba-4a5f-b73f-e63177585ff2","Type":"ContainerDied","Data":"b44c730079131456d7a678672fe7bd6c530a5e815b512532e5160aae053309b6"}
Nov 03 18:17:07.467642 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:07.458519    5315 scope.go:122] "RemoveContainer" containerID="67b2b09bdc9091fac78291aadbe5b9786299a10f208a965b93cbbeb07130fb3e"
Nov 03 18:17:07.467642 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:07.458427    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-232f9e-75b58576ff-2z4jb"
Nov 03 18:17:07.467642 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:07.461399    5315 generic.go:356] "Generic (PLEG): container finished" podID="c6ca8679-f0c2-48a0-8786-ed4cbad5295d" containerID="8366f8c386c20c9a31fd816b8f379cabd66ae38b01939c41eb7ea15da0cfaab6" exitCode=0
Nov 03 18:17:07.467642 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:07.461493    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-232f9e-649f6fd75-thwmg" event={"ID":"c6ca8679-f0c2-48a0-8786-ed4cbad5295d","Type":"ContainerDied","Data":"8366f8c386c20c9a31fd816b8f379cabd66ae38b01939c41eb7ea15da0cfaab6"}
Nov 03 18:17:07.469535 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:07.469507    5315 scope.go:122] "RemoveContainer" containerID="67b2b09bdc9091fac78291aadbe5b9786299a10f208a965b93cbbeb07130fb3e"
Nov 03 18:17:07.469869 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:17:07.469843    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67b2b09bdc9091fac78291aadbe5b9786299a10f208a965b93cbbeb07130fb3e\": container with ID starting with 67b2b09bdc9091fac78291aadbe5b9786299a10f208a965b93cbbeb07130fb3e not found: ID does not exist" containerID="67b2b09bdc9091fac78291aadbe5b9786299a10f208a965b93cbbeb07130fb3e"
Nov 03 18:17:07.469967 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:07.469879    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67b2b09bdc9091fac78291aadbe5b9786299a10f208a965b93cbbeb07130fb3e"} err="failed to get container status \"67b2b09bdc9091fac78291aadbe5b9786299a10f208a965b93cbbeb07130fb3e\": rpc error: code = NotFound desc = could not find container \"67b2b09bdc9091fac78291aadbe5b9786299a10f208a965b93cbbeb07130fb3e\": container with ID starting with 67b2b09bdc9091fac78291aadbe5b9786299a10f208a965b93cbbeb07130fb3e not found: ID does not exist"
Nov 03 18:17:07.523663 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:07.523627    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a25b9cd-06ba-4a5f-b73f-e63177585ff2-kube-api-access-6s6g9\" (UniqueName: \"kubernetes.io/projected/0a25b9cd-06ba-4a5f-b73f-e63177585ff2-kube-api-access-6s6g9\") pod \"0a25b9cd-06ba-4a5f-b73f-e63177585ff2\" (UID: \"0a25b9cd-06ba-4a5f-b73f-e63177585ff2\") "
Nov 03 18:17:07.523860 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:07.523683    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a25b9cd-06ba-4a5f-b73f-e63177585ff2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a25b9cd-06ba-4a5f-b73f-e63177585ff2-plugins-config-volume\") pod \"0a25b9cd-06ba-4a5f-b73f-e63177585ff2\" (UID: \"0a25b9cd-06ba-4a5f-b73f-e63177585ff2\") "
Nov 03 18:17:07.524110 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:07.524079    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a25b9cd-06ba-4a5f-b73f-e63177585ff2-plugins-config-volume" pod "0a25b9cd-06ba-4a5f-b73f-e63177585ff2" (UID: "0a25b9cd-06ba-4a5f-b73f-e63177585ff2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:17:07.524240 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:07.524192    5315 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a25b9cd-06ba-4a5f-b73f-e63177585ff2-plugins-config-volume\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:07.525828 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:07.525806    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a25b9cd-06ba-4a5f-b73f-e63177585ff2-kube-api-access-6s6g9" pod "0a25b9cd-06ba-4a5f-b73f-e63177585ff2" (UID: "0a25b9cd-06ba-4a5f-b73f-e63177585ff2"). InnerVolumeSpecName "kube-api-access-6s6g9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:17:07.542953 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:07.542928    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-232f9e-649f6fd75-thwmg"
Nov 03 18:17:07.625017 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:07.624975    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6ca8679-f0c2-48a0-8786-ed4cbad5295d-kube-api-access-54n6g\" (UniqueName: \"kubernetes.io/projected/c6ca8679-f0c2-48a0-8786-ed4cbad5295d-kube-api-access-54n6g\") pod \"c6ca8679-f0c2-48a0-8786-ed4cbad5295d\" (UID: \"c6ca8679-f0c2-48a0-8786-ed4cbad5295d\") "
Nov 03 18:17:07.625270 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:07.625072    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6ca8679-f0c2-48a0-8786-ed4cbad5295d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6ca8679-f0c2-48a0-8786-ed4cbad5295d-plugins-config-volume\") pod \"c6ca8679-f0c2-48a0-8786-ed4cbad5295d\" (UID: \"c6ca8679-f0c2-48a0-8786-ed4cbad5295d\") "
Nov 03 18:17:07.625462 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:07.625442    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6s6g9\" (UniqueName: \"kubernetes.io/projected/0a25b9cd-06ba-4a5f-b73f-e63177585ff2-kube-api-access-6s6g9\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:07.625531 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:07.625460    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6ca8679-f0c2-48a0-8786-ed4cbad5295d-plugins-config-volume" pod "c6ca8679-f0c2-48a0-8786-ed4cbad5295d" (UID: "c6ca8679-f0c2-48a0-8786-ed4cbad5295d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:17:07.627181 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:07.627161    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6ca8679-f0c2-48a0-8786-ed4cbad5295d-kube-api-access-54n6g" pod "c6ca8679-f0c2-48a0-8786-ed4cbad5295d" (UID: "c6ca8679-f0c2-48a0-8786-ed4cbad5295d"). InnerVolumeSpecName "kube-api-access-54n6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:17:07.726291 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:07.726207    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54n6g\" (UniqueName: \"kubernetes.io/projected/c6ca8679-f0c2-48a0-8786-ed4cbad5295d-kube-api-access-54n6g\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:07.726291 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:07.726235    5315 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6ca8679-f0c2-48a0-8786-ed4cbad5295d-plugins-config-volume\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:07.871915 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:07.871879    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-232f9e-75b58576ff-2z4jb"]
Nov 03 18:17:07.878166 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:07.878133    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-232f9e-75b58576ff-2z4jb"]
Nov 03 18:17:08.469769 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:08.469680    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-232f9e-649f6fd75-thwmg"
Nov 03 18:17:08.469769 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:08.469679    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-232f9e-649f6fd75-thwmg" event={"ID":"c6ca8679-f0c2-48a0-8786-ed4cbad5295d","Type":"ContainerDied","Data":"ad0103ce399d3f5719ab60c8dbc51e62cbe461078fb4512bd3ef32289989976b"}
Nov 03 18:17:08.470318 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:08.469796    5315 scope.go:122] "RemoveContainer" containerID="8366f8c386c20c9a31fd816b8f379cabd66ae38b01939c41eb7ea15da0cfaab6"
Nov 03 18:17:08.622497 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:08.622459    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-232f9e-649f6fd75-thwmg"]
Nov 03 18:17:08.668551 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:08.668517    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-232f9e-649f6fd75-thwmg"]
Nov 03 18:17:09.103988 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:09.103954    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a25b9cd-06ba-4a5f-b73f-e63177585ff2" path="/var/lib/kubelet/pods/0a25b9cd-06ba-4a5f-b73f-e63177585ff2/volumes"
Nov 03 18:17:09.104375 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:09.104361    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6ca8679-f0c2-48a0-8786-ed4cbad5295d" path="/var/lib/kubelet/pods/c6ca8679-f0c2-48a0-8786-ed4cbad5295d/volumes"
Nov 03 18:17:09.418955 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:09.418869    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f79c02-5477cd8fff-2h4nk" podUID="13fcfa33-03ce-4384-a431-edf17a1f2735" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:17:09.419114 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:09.418872    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-f79c02-5c7b8dcbf7-qtwl6" podUID="7861e9eb-a526-442b-800f-6009fa55789a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:17:09.917000 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:09.916966    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cac8e02-r44sw"]
Nov 03 18:17:09.917430 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:09.917416    5315 cpu_manager.go:414] "containerMap: removing container" podUID="6c42d4a5-6add-4665-ac0f-0a9ea35c9158" containerName="revoke-keys"
Nov 03 18:17:09.917477 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:09.917431    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c42d4a5-6add-4665-ac0f-0a9ea35c9158" containerName="revoke-keys"
Nov 03 18:17:09.917477 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:09.917454    5315 cpu_manager.go:414] "containerMap: removing container" podUID="0a25b9cd-06ba-4a5f-b73f-e63177585ff2" containerName="payload-processing"
Nov 03 18:17:09.917477 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:09.917460    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a25b9cd-06ba-4a5f-b73f-e63177585ff2" containerName="payload-processing"
Nov 03 18:17:09.917477 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:09.917473    5315 cpu_manager.go:414] "containerMap: removing container" podUID="c974b0f1-e00b-4e4a-b193-df769068983d" containerName="main"
Nov 03 18:17:09.917627 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:09.917479    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c974b0f1-e00b-4e4a-b193-df769068983d" containerName="main"
Nov 03 18:17:09.917627 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:09.917489    5315 cpu_manager.go:414] "containerMap: removing container" podUID="7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625" containerName="main"
Nov 03 18:17:09.917627 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:09.917503    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625" containerName="main"
Nov 03 18:17:09.917627 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:09.917513    5315 cpu_manager.go:414] "containerMap: removing container" podUID="36d4f867-6750-43de-8cdc-76f7e76fdd44" containerName="istio-proxy"
Nov 03 18:17:09.917627 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:09.917518    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36d4f867-6750-43de-8cdc-76f7e76fdd44" containerName="istio-proxy"
Nov 03 18:17:09.917627 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:09.917529    5315 cpu_manager.go:414] "containerMap: removing container" podUID="c6ca8679-f0c2-48a0-8786-ed4cbad5295d" containerName="payload-pre-processing"
Nov 03 18:17:09.917627 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:09.917534    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6ca8679-f0c2-48a0-8786-ed4cbad5295d" containerName="payload-pre-processing"
Nov 03 18:17:09.917627 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:09.917544    5315 cpu_manager.go:414] "containerMap: removing container" podUID="36d4f867-6750-43de-8cdc-76f7e76fdd44" containerName="istio-proxy"
Nov 03 18:17:09.917627 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:09.917549    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36d4f867-6750-43de-8cdc-76f7e76fdd44" containerName="istio-proxy"
Nov 03 18:17:09.917900 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:09.917633    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="c974b0f1-e00b-4e4a-b193-df769068983d" containerName="main"
Nov 03 18:17:09.917900 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:09.917645    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d3e8590-15f3-4ca2-ba34-9a8a9d0a9625" containerName="main"
Nov 03 18:17:09.917900 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:09.917652    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="36d4f867-6750-43de-8cdc-76f7e76fdd44" containerName="istio-proxy"
Nov 03 18:17:09.917900 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:09.917659    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6ca8679-f0c2-48a0-8786-ed4cbad5295d" containerName="payload-pre-processing"
Nov 03 18:17:09.917900 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:09.917666    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c42d4a5-6add-4665-ac0f-0a9ea35c9158" containerName="revoke-keys"
Nov 03 18:17:09.917900 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:09.917672    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="7296122e-d510-4826-9a6a-2f663d6603bf" containerName="istio-proxy"
Nov 03 18:17:09.917900 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:09.917679    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a25b9cd-06ba-4a5f-b73f-e63177585ff2" containerName="payload-processing"
Nov 03 18:17:09.922996 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:09.922972    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cac8e02-r44sw"
Nov 03 18:17:09.972733 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:09.972691    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1183615-p4qqj"]
Nov 03 18:17:09.973273 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:09.973256    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="36d4f867-6750-43de-8cdc-76f7e76fdd44" containerName="istio-proxy"
Nov 03 18:17:09.976669 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:09.976644    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1183615-p4qqj"
Nov 03 18:17:10.016413 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:10.016373    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:17:10.016413 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:10.016405    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:17:10.016702 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:10.016438    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:17:10.018755 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:10.018730    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:17:10.021304 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:10.021280    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cac8e02-r44sw"]
Nov 03 18:17:10.037819 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:10.037783    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1183615-p4qqj"]
Nov 03 18:17:10.057643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:10.057588    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/394f9a52-30a5-4902-bba9-7e9442e7a67f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1183615-p4qqj\" (UID: \"394f9a52-30a5-4902-bba9-7e9442e7a67f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1183615-p4qqj"
Nov 03 18:17:10.057825 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:10.057728    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19759972-5257-430b-abed-7a2f601b652b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cac8e02-r44sw\" (UID: \"19759972-5257-430b-abed-7a2f601b652b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cac8e02-r44sw"
Nov 03 18:17:10.057825 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:10.057780    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19759972-5257-430b-abed-7a2f601b652b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cac8e02-r44sw\" (UID: \"19759972-5257-430b-abed-7a2f601b652b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cac8e02-r44sw"
Nov 03 18:17:10.057825 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:10.057810    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tnhm\" (UniqueName: \"kubernetes.io/projected/394f9a52-30a5-4902-bba9-7e9442e7a67f-kube-api-access-2tnhm\") pod \"maas-api-revoke-sub-c1183615-p4qqj\" (UID: \"394f9a52-30a5-4902-bba9-7e9442e7a67f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1183615-p4qqj"
Nov 03 18:17:10.057969 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:10.057861    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/394f9a52-30a5-4902-bba9-7e9442e7a67f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1183615-p4qqj\" (UID: \"394f9a52-30a5-4902-bba9-7e9442e7a67f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1183615-p4qqj"
Nov 03 18:17:10.057969 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:10.057882    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw2xc\" (UniqueName: \"kubernetes.io/projected/19759972-5257-430b-abed-7a2f601b652b-kube-api-access-tw2xc\") pod \"maas-api-revoke-sub-0cac8e02-r44sw\" (UID: \"19759972-5257-430b-abed-7a2f601b652b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cac8e02-r44sw"
Nov 03 18:17:10.159486 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:10.159438    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/394f9a52-30a5-4902-bba9-7e9442e7a67f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1183615-p4qqj\" (UID: \"394f9a52-30a5-4902-bba9-7e9442e7a67f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1183615-p4qqj"
Nov 03 18:17:10.159740 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:10.159513    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19759972-5257-430b-abed-7a2f601b652b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cac8e02-r44sw\" (UID: \"19759972-5257-430b-abed-7a2f601b652b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cac8e02-r44sw"
Nov 03 18:17:10.159740 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:10.159544    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19759972-5257-430b-abed-7a2f601b652b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cac8e02-r44sw\" (UID: \"19759972-5257-430b-abed-7a2f601b652b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cac8e02-r44sw"
Nov 03 18:17:10.159740 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:10.159567    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tnhm\" (UniqueName: \"kubernetes.io/projected/394f9a52-30a5-4902-bba9-7e9442e7a67f-kube-api-access-2tnhm\") pod \"maas-api-revoke-sub-c1183615-p4qqj\" (UID: \"394f9a52-30a5-4902-bba9-7e9442e7a67f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1183615-p4qqj"
Nov 03 18:17:10.159740 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:10.159599    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/394f9a52-30a5-4902-bba9-7e9442e7a67f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1183615-p4qqj\" (UID: \"394f9a52-30a5-4902-bba9-7e9442e7a67f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1183615-p4qqj"
Nov 03 18:17:10.159740 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:10.159662    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw2xc\" (UniqueName: \"kubernetes.io/projected/19759972-5257-430b-abed-7a2f601b652b-kube-api-access-tw2xc\") pod \"maas-api-revoke-sub-0cac8e02-r44sw\" (UID: \"19759972-5257-430b-abed-7a2f601b652b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cac8e02-r44sw"
Nov 03 18:17:10.160363 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:10.160339    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/394f9a52-30a5-4902-bba9-7e9442e7a67f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1183615-p4qqj\" (UID: \"394f9a52-30a5-4902-bba9-7e9442e7a67f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1183615-p4qqj"
Nov 03 18:17:10.160500 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:10.160375    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19759972-5257-430b-abed-7a2f601b652b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cac8e02-r44sw\" (UID: \"19759972-5257-430b-abed-7a2f601b652b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cac8e02-r44sw"
Nov 03 18:17:10.162079 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:10.162056    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19759972-5257-430b-abed-7a2f601b652b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cac8e02-r44sw\" (UID: \"19759972-5257-430b-abed-7a2f601b652b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cac8e02-r44sw"
Nov 03 18:17:10.162079 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:10.162068    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/394f9a52-30a5-4902-bba9-7e9442e7a67f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1183615-p4qqj\" (UID: \"394f9a52-30a5-4902-bba9-7e9442e7a67f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1183615-p4qqj"
Nov 03 18:17:10.216837 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:10.216762    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw2xc\" (UniqueName: \"kubernetes.io/projected/19759972-5257-430b-abed-7a2f601b652b-kube-api-access-tw2xc\") pod \"maas-api-revoke-sub-0cac8e02-r44sw\" (UID: \"19759972-5257-430b-abed-7a2f601b652b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cac8e02-r44sw"
Nov 03 18:17:10.216984 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:10.216880    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tnhm\" (UniqueName: \"kubernetes.io/projected/394f9a52-30a5-4902-bba9-7e9442e7a67f-kube-api-access-2tnhm\") pod \"maas-api-revoke-sub-c1183615-p4qqj\" (UID: \"394f9a52-30a5-4902-bba9-7e9442e7a67f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1183615-p4qqj"
Nov 03 18:17:10.236666 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:10.236621    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cac8e02-r44sw"
Nov 03 18:17:10.291180 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:10.291145    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1183615-p4qqj"
Nov 03 18:17:10.607386 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:10.607345    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cac8e02-r44sw"]
Nov 03 18:17:10.613494 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:17:10.613466    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19759972_5257_430b_abed_7a2f601b652b.slice/crio-87e5bca9839e21f9bc08efacbfdea61f685a55866740c6ae3c728807e1b66aeb WatchSource:0}: Error finding container 87e5bca9839e21f9bc08efacbfdea61f685a55866740c6ae3c728807e1b66aeb: Status 404 returned error can't find the container with id 87e5bca9839e21f9bc08efacbfdea61f685a55866740c6ae3c728807e1b66aeb
Nov 03 18:17:10.710344 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:10.710304    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1183615-p4qqj"]
Nov 03 18:17:11.492716 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:11.492673    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1183615-p4qqj" event={"ID":"394f9a52-30a5-4902-bba9-7e9442e7a67f","Type":"ContainerStarted","Data":"7917c425f9e3d4a5150b30ba950781197c304c3ca867c326ad2bef7f4633dcf9"}
Nov 03 18:17:11.493207 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:11.492748    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1183615-p4qqj" event={"ID":"394f9a52-30a5-4902-bba9-7e9442e7a67f","Type":"ContainerStarted","Data":"c7807978e5fcc8fde4a36aff41750d611c1e1aa6b88d92cd735c33d1575c89f8"}
Nov 03 18:17:11.494322 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:11.494294    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cac8e02-r44sw" event={"ID":"19759972-5257-430b-abed-7a2f601b652b","Type":"ContainerStarted","Data":"51b47b66423189c87ca8b0529db3ed767577870cf470a63753f78bf89b83c631"}
Nov 03 18:17:11.494450 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:11.494343    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cac8e02-r44sw" event={"ID":"19759972-5257-430b-abed-7a2f601b652b","Type":"ContainerStarted","Data":"87e5bca9839e21f9bc08efacbfdea61f685a55866740c6ae3c728807e1b66aeb"}
Nov 03 18:17:11.602370 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:11.602312    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1183615-p4qqj" podStartSLOduration=2.602298433 podStartE2EDuration="2.602298433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:17:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:17:11.60022035 +0000 UTC m=+1870.616418416" watchObservedRunningTime="2026-11-03 18:17:11.602298433 +0000 UTC m=+1870.618496502"
Nov 03 18:17:11.712082 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:11.712033    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cac8e02-r44sw" podStartSLOduration=2.7120195860000003 podStartE2EDuration="2.712019586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:17:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:17:11.711533383 +0000 UTC m=+1870.727731452" watchObservedRunningTime="2026-11-03 18:17:11.712019586 +0000 UTC m=+1870.728217654"
Nov 03 18:17:12.502184 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:12.502152    5315 generic.go:356] "Generic (PLEG): container finished" podID="394f9a52-30a5-4902-bba9-7e9442e7a67f" containerID="7917c425f9e3d4a5150b30ba950781197c304c3ca867c326ad2bef7f4633dcf9" exitCode=0
Nov 03 18:17:12.502594 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:12.502236    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1183615-p4qqj" event={"ID":"394f9a52-30a5-4902-bba9-7e9442e7a67f","Type":"ContainerDied","Data":"7917c425f9e3d4a5150b30ba950781197c304c3ca867c326ad2bef7f4633dcf9"}
Nov 03 18:17:12.503841 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:12.503820    5315 generic.go:356] "Generic (PLEG): container finished" podID="19759972-5257-430b-abed-7a2f601b652b" containerID="51b47b66423189c87ca8b0529db3ed767577870cf470a63753f78bf89b83c631" exitCode=0
Nov 03 18:17:12.503894 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:12.503855    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cac8e02-r44sw" event={"ID":"19759972-5257-430b-abed-7a2f601b652b","Type":"ContainerDied","Data":"51b47b66423189c87ca8b0529db3ed767577870cf470a63753f78bf89b83c631"}
Nov 03 18:17:13.687957 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:13.687928    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1183615-p4qqj"
Nov 03 18:17:13.691134 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:13.691110    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cac8e02-r44sw"
Nov 03 18:17:13.803271 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:13.803237    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/394f9a52-30a5-4902-bba9-7e9442e7a67f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/394f9a52-30a5-4902-bba9-7e9442e7a67f-maas-api-xks-ca\") pod \"394f9a52-30a5-4902-bba9-7e9442e7a67f\" (UID: \"394f9a52-30a5-4902-bba9-7e9442e7a67f\") "
Nov 03 18:17:13.803470 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:13.803314    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/394f9a52-30a5-4902-bba9-7e9442e7a67f-kube-api-access-2tnhm\" (UniqueName: \"kubernetes.io/projected/394f9a52-30a5-4902-bba9-7e9442e7a67f-kube-api-access-2tnhm\") pod \"394f9a52-30a5-4902-bba9-7e9442e7a67f\" (UID: \"394f9a52-30a5-4902-bba9-7e9442e7a67f\") "
Nov 03 18:17:13.803470 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:13.803333    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19759972-5257-430b-abed-7a2f601b652b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19759972-5257-430b-abed-7a2f601b652b-maas-api-xks-ca\") pod \"19759972-5257-430b-abed-7a2f601b652b\" (UID: \"19759972-5257-430b-abed-7a2f601b652b\") "
Nov 03 18:17:13.803470 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:13.803365    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19759972-5257-430b-abed-7a2f601b652b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19759972-5257-430b-abed-7a2f601b652b-maas-api-openshift-ca\") pod \"19759972-5257-430b-abed-7a2f601b652b\" (UID: \"19759972-5257-430b-abed-7a2f601b652b\") "
Nov 03 18:17:13.803470 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:13.803421    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/394f9a52-30a5-4902-bba9-7e9442e7a67f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/394f9a52-30a5-4902-bba9-7e9442e7a67f-maas-api-openshift-ca\") pod \"394f9a52-30a5-4902-bba9-7e9442e7a67f\" (UID: \"394f9a52-30a5-4902-bba9-7e9442e7a67f\") "
Nov 03 18:17:13.803795 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:13.803486    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19759972-5257-430b-abed-7a2f601b652b-kube-api-access-tw2xc\" (UniqueName: \"kubernetes.io/projected/19759972-5257-430b-abed-7a2f601b652b-kube-api-access-tw2xc\") pod \"19759972-5257-430b-abed-7a2f601b652b\" (UID: \"19759972-5257-430b-abed-7a2f601b652b\") "
Nov 03 18:17:13.803795 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:13.803762    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19759972-5257-430b-abed-7a2f601b652b-maas-api-openshift-ca" pod "19759972-5257-430b-abed-7a2f601b652b" (UID: "19759972-5257-430b-abed-7a2f601b652b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:17:13.804033 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:13.803978    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19759972-5257-430b-abed-7a2f601b652b-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:13.804033 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:13.804009    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/394f9a52-30a5-4902-bba9-7e9442e7a67f-maas-api-openshift-ca" pod "394f9a52-30a5-4902-bba9-7e9442e7a67f" (UID: "394f9a52-30a5-4902-bba9-7e9442e7a67f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:17:13.806093 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:13.806063    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19759972-5257-430b-abed-7a2f601b652b-maas-api-xks-ca" pod "19759972-5257-430b-abed-7a2f601b652b" (UID: "19759972-5257-430b-abed-7a2f601b652b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:17:13.806232 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:13.806150    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/394f9a52-30a5-4902-bba9-7e9442e7a67f-kube-api-access-2tnhm" pod "394f9a52-30a5-4902-bba9-7e9442e7a67f" (UID: "394f9a52-30a5-4902-bba9-7e9442e7a67f"). InnerVolumeSpecName "kube-api-access-2tnhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:17:13.806370 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:13.806349    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/394f9a52-30a5-4902-bba9-7e9442e7a67f-maas-api-xks-ca" pod "394f9a52-30a5-4902-bba9-7e9442e7a67f" (UID: "394f9a52-30a5-4902-bba9-7e9442e7a67f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:17:13.806648 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:13.806624    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19759972-5257-430b-abed-7a2f601b652b-kube-api-access-tw2xc" pod "19759972-5257-430b-abed-7a2f601b652b" (UID: "19759972-5257-430b-abed-7a2f601b652b"). InnerVolumeSpecName "kube-api-access-tw2xc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:17:13.904963 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:13.904923    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/394f9a52-30a5-4902-bba9-7e9442e7a67f-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:13.904963 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:13.904962    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tnhm\" (UniqueName: \"kubernetes.io/projected/394f9a52-30a5-4902-bba9-7e9442e7a67f-kube-api-access-2tnhm\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:13.905187 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:13.904978    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19759972-5257-430b-abed-7a2f601b652b-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:13.905187 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:13.904991    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/394f9a52-30a5-4902-bba9-7e9442e7a67f-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:13.905187 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:13.905005    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw2xc\" (UniqueName: \"kubernetes.io/projected/19759972-5257-430b-abed-7a2f601b652b-kube-api-access-tw2xc\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:14.518750 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:14.518680    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1183615-p4qqj"
Nov 03 18:17:14.518966 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:14.518692    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1183615-p4qqj" event={"ID":"394f9a52-30a5-4902-bba9-7e9442e7a67f","Type":"ContainerDied","Data":"c7807978e5fcc8fde4a36aff41750d611c1e1aa6b88d92cd735c33d1575c89f8"}
Nov 03 18:17:14.518966 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:14.518810    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7807978e5fcc8fde4a36aff41750d611c1e1aa6b88d92cd735c33d1575c89f8"
Nov 03 18:17:14.520313 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:14.520294    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cac8e02-r44sw"
Nov 03 18:17:14.520427 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:14.520328    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cac8e02-r44sw" event={"ID":"19759972-5257-430b-abed-7a2f601b652b","Type":"ContainerDied","Data":"87e5bca9839e21f9bc08efacbfdea61f685a55866740c6ae3c728807e1b66aeb"}
Nov 03 18:17:14.520427 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:14.520364    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87e5bca9839e21f9bc08efacbfdea61f685a55866740c6ae3c728807e1b66aeb"
Nov 03 18:17:16.851340 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:16.851306    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-232f9e-h58gj"]
Nov 03 18:17:16.897351 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:16.897317    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-232f9e-h58gj"]
Nov 03 18:17:17.103459 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:17.103370    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c42d4a5-6add-4665-ac0f-0a9ea35c9158" path="/var/lib/kubelet/pods/6c42d4a5-6add-4665-ac0f-0a9ea35c9158/volumes"
Nov 03 18:17:17.791179 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:17.791144    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7338-5d5aed"]
Nov 03 18:17:17.791641 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:17.791627    5315 cpu_manager.go:414] "containerMap: removing container" podUID="19759972-5257-430b-abed-7a2f601b652b" containerName="revoke-keys"
Nov 03 18:17:17.791710 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:17.791642    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19759972-5257-430b-abed-7a2f601b652b" containerName="revoke-keys"
Nov 03 18:17:17.791710 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:17.791652    5315 cpu_manager.go:414] "containerMap: removing container" podUID="394f9a52-30a5-4902-bba9-7e9442e7a67f" containerName="revoke-keys"
Nov 03 18:17:17.791710 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:17.791658    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="394f9a52-30a5-4902-bba9-7e9442e7a67f" containerName="revoke-keys"
Nov 03 18:17:17.791810 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:17.791731    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="19759972-5257-430b-abed-7a2f601b652b" containerName="revoke-keys"
Nov 03 18:17:17.791810 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:17.791739    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="394f9a52-30a5-4902-bba9-7e9442e7a67f" containerName="revoke-keys"
Nov 03 18:17:17.796408 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:17.796380    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7338-5d5aed"
Nov 03 18:17:17.845052 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:17.845017    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-447v9\""
Nov 03 18:17:17.893153 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:17.893115    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7338-5d5aed"]
Nov 03 18:17:17.957543 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:17.957497    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rt8f\" (UniqueName: \"kubernetes.io/projected/8dc08ecb-c6d1-4db2-8170-ca4774a7d946-kube-api-access-5rt8f\") pod \"test-curl-7338-5d5aed\" (UID: \"8dc08ecb-c6d1-4db2-8170-ca4774a7d946\") " pod="opendatahub/test-curl-7338-5d5aed"
Nov 03 18:17:18.059448 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:18.059348    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rt8f\" (UniqueName: \"kubernetes.io/projected/8dc08ecb-c6d1-4db2-8170-ca4774a7d946-kube-api-access-5rt8f\") pod \"test-curl-7338-5d5aed\" (UID: \"8dc08ecb-c6d1-4db2-8170-ca4774a7d946\") " pod="opendatahub/test-curl-7338-5d5aed"
Nov 03 18:17:18.116288 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:18.116254    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rt8f\" (UniqueName: \"kubernetes.io/projected/8dc08ecb-c6d1-4db2-8170-ca4774a7d946-kube-api-access-5rt8f\") pod \"test-curl-7338-5d5aed\" (UID: \"8dc08ecb-c6d1-4db2-8170-ca4774a7d946\") " pod="opendatahub/test-curl-7338-5d5aed"
Nov 03 18:17:18.411914 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:18.411812    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7338-5d5aed"
Nov 03 18:17:18.920134 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:18.920022    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7338-5d5aed"]
Nov 03 18:17:19.419958 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:19.419912    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-f79c02-5c7b8dcbf7-qtwl6" podUID="7861e9eb-a526-442b-800f-6009fa55789a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:17:19.420148 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:19.419921    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f79c02-5477cd8fff-2h4nk" podUID="13fcfa33-03ce-4384-a431-edf17a1f2735" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:17:19.585556 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:19.585510    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7338-5d5aed" event={"ID":"8dc08ecb-c6d1-4db2-8170-ca4774a7d946","Type":"ContainerStarted","Data":"ede507ae752da04b0b1ae794492367c93999e0ef8d15a2fe00b64cba73f8cae8"}
Nov 03 18:17:19.585757 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:19.585570    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7338-5d5aed" event={"ID":"8dc08ecb-c6d1-4db2-8170-ca4774a7d946","Type":"ContainerStarted","Data":"1606029ec35b0779e2c56442fb9819d9b776195db18b69ee0de3f884305bcbe3"}
Nov 03 18:17:19.693662 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:19.693545    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7338-5d5aed" podStartSLOduration=2.69353175 podStartE2EDuration="2.69353175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:17:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:17:19.691855552 +0000 UTC m=+1878.708053621" watchObservedRunningTime="2026-11-03 18:17:19.69353175 +0000 UTC m=+1878.709729809"
Nov 03 18:17:21.232224 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:21.232188    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7338-5d5aed"]
Nov 03 18:17:21.236575 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:21.236543    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7338-5d5aed"]
Nov 03 18:17:21.606645 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:21.605349    5315 generic.go:356] "Generic (PLEG): container finished" podID="13fcfa33-03ce-4384-a431-edf17a1f2735" containerID="da8194e643108ce1d44254e047b5d88c793a05b242660947f1578a1fd2af28ba" exitCode=0
Nov 03 18:17:21.606645 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:21.605408    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f79c02-5477cd8fff-2h4nk" event={"ID":"13fcfa33-03ce-4384-a431-edf17a1f2735","Type":"ContainerDied","Data":"da8194e643108ce1d44254e047b5d88c793a05b242660947f1578a1fd2af28ba"}
Nov 03 18:17:21.607666 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:21.607636    5315 generic.go:356] "Generic (PLEG): container finished" podID="7861e9eb-a526-442b-800f-6009fa55789a" containerID="cb5b039bde5463d14872c292f2434ff1fc45bd3258d6b823d2919b57416e6042" exitCode=0
Nov 03 18:17:21.607818 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:21.607720    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-f79c02-5c7b8dcbf7-qtwl6" event={"ID":"7861e9eb-a526-442b-800f-6009fa55789a","Type":"ContainerDied","Data":"cb5b039bde5463d14872c292f2434ff1fc45bd3258d6b823d2919b57416e6042"}
Nov 03 18:17:21.607818 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:21.607803    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7338-5d5aed" podUID="8dc08ecb-c6d1-4db2-8170-ca4774a7d946" containerName="test-curl-7338-5d5aed" containerID="cri-o://ede507ae752da04b0b1ae794492367c93999e0ef8d15a2fe00b64cba73f8cae8" gracePeriod=2
Nov 03 18:17:21.641674 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:21.641646    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-f79c02-5c7b8dcbf7-qtwl6"
Nov 03 18:17:21.655116 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:17:21.655079    5315 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7338-5d5aed\" is forbidden: User \"system:node:ip-10-0-128-204.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-204.us-west-2.compute.internal' and this object" podUID="8dc08ecb-c6d1-4db2-8170-ca4774a7d946" pod="opendatahub/test-curl-7338-5d5aed"
Nov 03 18:17:21.681177 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:21.681150    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f79c02-5477cd8fff-2h4nk"
Nov 03 18:17:21.736200 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:21.736114    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7861e9eb-a526-442b-800f-6009fa55789a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7861e9eb-a526-442b-800f-6009fa55789a-plugins-config-volume\") pod \"7861e9eb-a526-442b-800f-6009fa55789a\" (UID: \"7861e9eb-a526-442b-800f-6009fa55789a\") "
Nov 03 18:17:21.736388 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:21.736271    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7861e9eb-a526-442b-800f-6009fa55789a-kube-api-access-2k6hx\" (UniqueName: \"kubernetes.io/projected/7861e9eb-a526-442b-800f-6009fa55789a-kube-api-access-2k6hx\") pod \"7861e9eb-a526-442b-800f-6009fa55789a\" (UID: \"7861e9eb-a526-442b-800f-6009fa55789a\") "
Nov 03 18:17:21.736388 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:21.736332    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13fcfa33-03ce-4384-a431-edf17a1f2735-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13fcfa33-03ce-4384-a431-edf17a1f2735-plugins-config-volume\") pod \"13fcfa33-03ce-4384-a431-edf17a1f2735\" (UID: \"13fcfa33-03ce-4384-a431-edf17a1f2735\") "
Nov 03 18:17:21.736388 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:21.736353    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13fcfa33-03ce-4384-a431-edf17a1f2735-kube-api-access-sxh6d\" (UniqueName: \"kubernetes.io/projected/13fcfa33-03ce-4384-a431-edf17a1f2735-kube-api-access-sxh6d\") pod \"13fcfa33-03ce-4384-a431-edf17a1f2735\" (UID: \"13fcfa33-03ce-4384-a431-edf17a1f2735\") "
Nov 03 18:17:21.736719 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:21.736691    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7861e9eb-a526-442b-800f-6009fa55789a-plugins-config-volume" pod "7861e9eb-a526-442b-800f-6009fa55789a" (UID: "7861e9eb-a526-442b-800f-6009fa55789a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:17:21.736842 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:21.736746    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13fcfa33-03ce-4384-a431-edf17a1f2735-plugins-config-volume" pod "13fcfa33-03ce-4384-a431-edf17a1f2735" (UID: "13fcfa33-03ce-4384-a431-edf17a1f2735"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:17:21.738539 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:21.738521    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7861e9eb-a526-442b-800f-6009fa55789a-kube-api-access-2k6hx" pod "7861e9eb-a526-442b-800f-6009fa55789a" (UID: "7861e9eb-a526-442b-800f-6009fa55789a"). InnerVolumeSpecName "kube-api-access-2k6hx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:17:21.738648 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:21.738596    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13fcfa33-03ce-4384-a431-edf17a1f2735-kube-api-access-sxh6d" pod "13fcfa33-03ce-4384-a431-edf17a1f2735" (UID: "13fcfa33-03ce-4384-a431-edf17a1f2735"). InnerVolumeSpecName "kube-api-access-sxh6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:17:21.811601 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:17:21.811554    5315 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7338-5d5aed\" is forbidden: User \"system:node:ip-10-0-128-204.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-204.us-west-2.compute.internal' and this object" podUID="8dc08ecb-c6d1-4db2-8170-ca4774a7d946" pod="opendatahub/test-curl-7338-5d5aed"
Nov 03 18:17:21.818161 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:21.818124    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1720c00c-24tp8"]
Nov 03 18:17:21.818652 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:21.818634    5315 cpu_manager.go:414] "containerMap: removing container" podUID="8dc08ecb-c6d1-4db2-8170-ca4774a7d946" containerName="test-curl-7338-5d5aed"
Nov 03 18:17:21.818756 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:21.818652    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dc08ecb-c6d1-4db2-8170-ca4774a7d946" containerName="test-curl-7338-5d5aed"
Nov 03 18:17:21.818756 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:21.818676    5315 cpu_manager.go:414] "containerMap: removing container" podUID="13fcfa33-03ce-4384-a431-edf17a1f2735" containerName="payload-pre-processing"
Nov 03 18:17:21.818756 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:21.818683    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13fcfa33-03ce-4384-a431-edf17a1f2735" containerName="payload-pre-processing"
Nov 03 18:17:21.818756 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:21.818693    5315 cpu_manager.go:414] "containerMap: removing container" podUID="7861e9eb-a526-442b-800f-6009fa55789a" containerName="payload-processing"
Nov 03 18:17:21.818756 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:21.818698    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7861e9eb-a526-442b-800f-6009fa55789a" containerName="payload-processing"
Nov 03 18:17:21.818942 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:21.818768    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dc08ecb-c6d1-4db2-8170-ca4774a7d946" containerName="test-curl-7338-5d5aed"
Nov 03 18:17:21.818942 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:21.818779    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="13fcfa33-03ce-4384-a431-edf17a1f2735" containerName="payload-pre-processing"
Nov 03 18:17:21.818942 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:21.818786    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="7861e9eb-a526-442b-800f-6009fa55789a" containerName="payload-processing"
Nov 03 18:17:21.822211 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:21.822189    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1720c00c-24tp8"
Nov 03 18:17:21.837499 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:21.837472    5315 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7861e9eb-a526-442b-800f-6009fa55789a-plugins-config-volume\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:21.837622 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:21.837502    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k6hx\" (UniqueName: \"kubernetes.io/projected/7861e9eb-a526-442b-800f-6009fa55789a-kube-api-access-2k6hx\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:21.837622 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:21.837517    5315 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13fcfa33-03ce-4384-a431-edf17a1f2735-plugins-config-volume\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:21.837622 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:21.837535    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxh6d\" (UniqueName: \"kubernetes.io/projected/13fcfa33-03ce-4384-a431-edf17a1f2735-kube-api-access-sxh6d\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:21.858941 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:17:21.858902    5315 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7338-5d5aed\" is forbidden: User \"system:node:ip-10-0-128-204.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-204.us-west-2.compute.internal' and this object" podUID="8dc08ecb-c6d1-4db2-8170-ca4774a7d946" pod="opendatahub/test-curl-7338-5d5aed"
Nov 03 18:17:21.870722 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:21.870691    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:17:21.870877 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:21.870850    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:17:21.870877 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:21.870850    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:17:21.870986 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:21.870914    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:17:21.883811 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:21.883777    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1720c00c-24tp8"]
Nov 03 18:17:21.938125 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:21.938090    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx65l\" (UniqueName: \"kubernetes.io/projected/abd83ea1-a9ec-42f3-a1f1-2943ff79d809-kube-api-access-mx65l\") pod \"maas-api-revoke-keys-e2e-mt-1720c00c-24tp8\" (UID: \"abd83ea1-a9ec-42f3-a1f1-2943ff79d809\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1720c00c-24tp8"
Nov 03 18:17:21.938125 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:21.938130    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abd83ea1-a9ec-42f3-a1f1-2943ff79d809-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-1720c00c-24tp8\" (UID: \"abd83ea1-a9ec-42f3-a1f1-2943ff79d809\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1720c00c-24tp8"
Nov 03 18:17:21.938353 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:21.938242    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abd83ea1-a9ec-42f3-a1f1-2943ff79d809-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-1720c00c-24tp8\" (UID: \"abd83ea1-a9ec-42f3-a1f1-2943ff79d809\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1720c00c-24tp8"
Nov 03 18:17:22.039139 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:22.039102    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx65l\" (UniqueName: \"kubernetes.io/projected/abd83ea1-a9ec-42f3-a1f1-2943ff79d809-kube-api-access-mx65l\") pod \"maas-api-revoke-keys-e2e-mt-1720c00c-24tp8\" (UID: \"abd83ea1-a9ec-42f3-a1f1-2943ff79d809\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1720c00c-24tp8"
Nov 03 18:17:22.039322 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:22.039144    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abd83ea1-a9ec-42f3-a1f1-2943ff79d809-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-1720c00c-24tp8\" (UID: \"abd83ea1-a9ec-42f3-a1f1-2943ff79d809\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1720c00c-24tp8"
Nov 03 18:17:22.039322 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:22.039196    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abd83ea1-a9ec-42f3-a1f1-2943ff79d809-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-1720c00c-24tp8\" (UID: \"abd83ea1-a9ec-42f3-a1f1-2943ff79d809\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1720c00c-24tp8"
Nov 03 18:17:22.039871 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:22.039840    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abd83ea1-a9ec-42f3-a1f1-2943ff79d809-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-1720c00c-24tp8\" (UID: \"abd83ea1-a9ec-42f3-a1f1-2943ff79d809\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1720c00c-24tp8"
Nov 03 18:17:22.041620 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:22.041581    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abd83ea1-a9ec-42f3-a1f1-2943ff79d809-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-1720c00c-24tp8\" (UID: \"abd83ea1-a9ec-42f3-a1f1-2943ff79d809\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1720c00c-24tp8"
Nov 03 18:17:22.096038 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:22.096002    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx65l\" (UniqueName: \"kubernetes.io/projected/abd83ea1-a9ec-42f3-a1f1-2943ff79d809-kube-api-access-mx65l\") pod \"maas-api-revoke-keys-e2e-mt-1720c00c-24tp8\" (UID: \"abd83ea1-a9ec-42f3-a1f1-2943ff79d809\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1720c00c-24tp8"
Nov 03 18:17:22.123295 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:17:22.123183    5315 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7338-5d5aed\" is forbidden: User \"system:node:ip-10-0-128-204.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-204.us-west-2.compute.internal' and this object" podUID="8dc08ecb-c6d1-4db2-8170-ca4774a7d946" pod="opendatahub/test-curl-7338-5d5aed"
Nov 03 18:17:22.135407 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:22.135378    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1720c00c-24tp8"
Nov 03 18:17:22.508046 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:22.508006    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1720c00c-24tp8"]
Nov 03 18:17:22.514105 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:17:22.514073    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabd83ea1_a9ec_42f3_a1f1_2943ff79d809.slice/crio-2776d90a5037d9b0cdc3524e5bea89faa3f8e64eced66f566618854cc34e64d3 WatchSource:0}: Error finding container 2776d90a5037d9b0cdc3524e5bea89faa3f8e64eced66f566618854cc34e64d3: Status 404 returned error can't find the container with id 2776d90a5037d9b0cdc3524e5bea89faa3f8e64eced66f566618854cc34e64d3
Nov 03 18:17:22.615529 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:22.615446    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f79c02-5477cd8fff-2h4nk"
Nov 03 18:17:22.615529 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:22.615463    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f79c02-5477cd8fff-2h4nk" event={"ID":"13fcfa33-03ce-4384-a431-edf17a1f2735","Type":"ContainerDied","Data":"0979ea834b2da0709685b58a011305ba3a956c4f126c7cafdba7608f906064bb"}
Nov 03 18:17:22.615783 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:22.615533    5315 scope.go:122] "RemoveContainer" containerID="da8194e643108ce1d44254e047b5d88c793a05b242660947f1578a1fd2af28ba"
Nov 03 18:17:22.617015 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:22.616985    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-f79c02-5c7b8dcbf7-qtwl6" event={"ID":"7861e9eb-a526-442b-800f-6009fa55789a","Type":"ContainerDied","Data":"bbaf8e680ee250859dde02a01d2e13d78a08b2583f93b3c39e00a16935010d61"}
Nov 03 18:17:22.617137 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:22.617020    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-f79c02-5c7b8dcbf7-qtwl6"
Nov 03 18:17:22.618577 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:22.618551    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1720c00c-24tp8" event={"ID":"abd83ea1-a9ec-42f3-a1f1-2943ff79d809","Type":"ContainerStarted","Data":"2776d90a5037d9b0cdc3524e5bea89faa3f8e64eced66f566618854cc34e64d3"}
Nov 03 18:17:22.625886 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:22.625865    5315 scope.go:122] "RemoveContainer" containerID="cb5b039bde5463d14872c292f2434ff1fc45bd3258d6b823d2919b57416e6042"
Nov 03 18:17:22.712111 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:17:22.712064    5315 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7338-5d5aed\" is forbidden: User \"system:node:ip-10-0-128-204.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-204.us-west-2.compute.internal' and this object" podUID="8dc08ecb-c6d1-4db2-8170-ca4774a7d946" pod="opendatahub/test-curl-7338-5d5aed"
Nov 03 18:17:22.815522 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:22.815480    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-f79c02-5477cd8fff-2h4nk"]
Nov 03 18:17:22.821698 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:22.821666    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-f79c02-5477cd8fff-2h4nk"]
Nov 03 18:17:22.922847 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:22.922751    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-f79c02-5c7b8dcbf7-qtwl6"]
Nov 03 18:17:22.926501 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:22.926471    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-f79c02-5c7b8dcbf7-qtwl6"]
Nov 03 18:17:22.971173 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:17:22.971130    5315 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7338-5d5aed\" is forbidden: User \"system:node:ip-10-0-128-204.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-204.us-west-2.compute.internal' and this object" podUID="8dc08ecb-c6d1-4db2-8170-ca4774a7d946" pod="opendatahub/test-curl-7338-5d5aed"
Nov 03 18:17:23.103400 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:23.103364    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13fcfa33-03ce-4384-a431-edf17a1f2735" path="/var/lib/kubelet/pods/13fcfa33-03ce-4384-a431-edf17a1f2735/volumes"
Nov 03 18:17:23.103810 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:23.103796    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7861e9eb-a526-442b-800f-6009fa55789a" path="/var/lib/kubelet/pods/7861e9eb-a526-442b-800f-6009fa55789a/volumes"
Nov 03 18:17:23.628921 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:23.628892    5315 generic.go:356] "Generic (PLEG): container finished" podID="abd83ea1-a9ec-42f3-a1f1-2943ff79d809" containerID="568cabb51e7caf5bde5c2b68179ed7500cac1ec0f8a69a199de7625809a95cf2" exitCode=0
Nov 03 18:17:23.629219 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:23.628972    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1720c00c-24tp8" event={"ID":"abd83ea1-a9ec-42f3-a1f1-2943ff79d809","Type":"ContainerDied","Data":"568cabb51e7caf5bde5c2b68179ed7500cac1ec0f8a69a199de7625809a95cf2"}
Nov 03 18:17:23.842863 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:23.842835    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7338-5d5aed"
Nov 03 18:17:23.961232 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:23.961145    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dc08ecb-c6d1-4db2-8170-ca4774a7d946-kube-api-access-5rt8f\" (UniqueName: \"kubernetes.io/projected/8dc08ecb-c6d1-4db2-8170-ca4774a7d946-kube-api-access-5rt8f\") pod \"8dc08ecb-c6d1-4db2-8170-ca4774a7d946\" (UID: \"8dc08ecb-c6d1-4db2-8170-ca4774a7d946\") "
Nov 03 18:17:23.963309 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:23.963275    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dc08ecb-c6d1-4db2-8170-ca4774a7d946-kube-api-access-5rt8f" pod "8dc08ecb-c6d1-4db2-8170-ca4774a7d946" (UID: "8dc08ecb-c6d1-4db2-8170-ca4774a7d946"). InnerVolumeSpecName "kube-api-access-5rt8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:17:24.063150 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:24.063108    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rt8f\" (UniqueName: \"kubernetes.io/projected/8dc08ecb-c6d1-4db2-8170-ca4774a7d946-kube-api-access-5rt8f\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:24.474153 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:24.474114    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g"]
Nov 03 18:17:24.474374 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:24.474347    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g" podUID="915e71a9-f7c9-4be4-929a-af6be579b6bd" containerName="istio-proxy" containerID="cri-o://8f4337e2ad655e0367f10693fec2249a1c5648782bc150d4bae14cad4d970281" gracePeriod=30
Nov 03 18:17:24.644215 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:24.643852    5315 generic.go:356] "Generic (PLEG): container finished" podID="8dc08ecb-c6d1-4db2-8170-ca4774a7d946" containerID="ede507ae752da04b0b1ae794492367c93999e0ef8d15a2fe00b64cba73f8cae8" exitCode=137
Nov 03 18:17:24.644215 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:24.644197    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7338-5d5aed"
Nov 03 18:17:24.651271 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:24.644808    5315 scope.go:122] "RemoveContainer" containerID="ede507ae752da04b0b1ae794492367c93999e0ef8d15a2fe00b64cba73f8cae8"
Nov 03 18:17:24.662775 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:17:24.662735    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:17:24.664874 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:24.663303    5315 scope.go:122] "RemoveContainer" containerID="ede507ae752da04b0b1ae794492367c93999e0ef8d15a2fe00b64cba73f8cae8"
Nov 03 18:17:24.667685 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:17:24.667179    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ede507ae752da04b0b1ae794492367c93999e0ef8d15a2fe00b64cba73f8cae8\": container with ID starting with ede507ae752da04b0b1ae794492367c93999e0ef8d15a2fe00b64cba73f8cae8 not found: ID does not exist" containerID="ede507ae752da04b0b1ae794492367c93999e0ef8d15a2fe00b64cba73f8cae8"
Nov 03 18:17:24.667685 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:24.667223    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ede507ae752da04b0b1ae794492367c93999e0ef8d15a2fe00b64cba73f8cae8"} err="failed to get container status \"ede507ae752da04b0b1ae794492367c93999e0ef8d15a2fe00b64cba73f8cae8\": rpc error: code = NotFound desc = could not find container \"ede507ae752da04b0b1ae794492367c93999e0ef8d15a2fe00b64cba73f8cae8\": container with ID starting with ede507ae752da04b0b1ae794492367c93999e0ef8d15a2fe00b64cba73f8cae8 not found: ID does not exist"
Nov 03 18:17:24.713795 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:17:24.713755    5315 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7338-5d5aed\" is forbidden: User \"system:node:ip-10-0-128-204.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-204.us-west-2.compute.internal' and this object" podUID="8dc08ecb-c6d1-4db2-8170-ca4774a7d946" pod="opendatahub/test-curl-7338-5d5aed"
Nov 03 18:17:24.802911 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:24.802880    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1720c00c-24tp8"
Nov 03 18:17:24.874771 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:24.874730    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abd83ea1-a9ec-42f3-a1f1-2943ff79d809-kube-api-access-mx65l\" (UniqueName: \"kubernetes.io/projected/abd83ea1-a9ec-42f3-a1f1-2943ff79d809-kube-api-access-mx65l\") pod \"abd83ea1-a9ec-42f3-a1f1-2943ff79d809\" (UID: \"abd83ea1-a9ec-42f3-a1f1-2943ff79d809\") "
Nov 03 18:17:24.874977 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:24.874828    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abd83ea1-a9ec-42f3-a1f1-2943ff79d809-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abd83ea1-a9ec-42f3-a1f1-2943ff79d809-maas-api-openshift-ca\") pod \"abd83ea1-a9ec-42f3-a1f1-2943ff79d809\" (UID: \"abd83ea1-a9ec-42f3-a1f1-2943ff79d809\") "
Nov 03 18:17:24.874977 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:24.874918    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abd83ea1-a9ec-42f3-a1f1-2943ff79d809-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abd83ea1-a9ec-42f3-a1f1-2943ff79d809-maas-api-xks-ca\") pod \"abd83ea1-a9ec-42f3-a1f1-2943ff79d809\" (UID: \"abd83ea1-a9ec-42f3-a1f1-2943ff79d809\") "
Nov 03 18:17:24.875203 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:24.875179    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abd83ea1-a9ec-42f3-a1f1-2943ff79d809-maas-api-openshift-ca" pod "abd83ea1-a9ec-42f3-a1f1-2943ff79d809" (UID: "abd83ea1-a9ec-42f3-a1f1-2943ff79d809"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:17:24.875378 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:24.875358    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abd83ea1-a9ec-42f3-a1f1-2943ff79d809-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:24.877046 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:24.877014    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abd83ea1-a9ec-42f3-a1f1-2943ff79d809-maas-api-xks-ca" pod "abd83ea1-a9ec-42f3-a1f1-2943ff79d809" (UID: "abd83ea1-a9ec-42f3-a1f1-2943ff79d809"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:17:24.877156 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:24.877013    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abd83ea1-a9ec-42f3-a1f1-2943ff79d809-kube-api-access-mx65l" pod "abd83ea1-a9ec-42f3-a1f1-2943ff79d809" (UID: "abd83ea1-a9ec-42f3-a1f1-2943ff79d809"). InnerVolumeSpecName "kube-api-access-mx65l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:17:24.957528 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:17:24.957481    5315 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7338-5d5aed\" is forbidden: User \"system:node:ip-10-0-128-204.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-204.us-west-2.compute.internal' and this object" podUID="8dc08ecb-c6d1-4db2-8170-ca4774a7d946" pod="opendatahub/test-curl-7338-5d5aed"
Nov 03 18:17:24.976289 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:24.976257    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abd83ea1-a9ec-42f3-a1f1-2943ff79d809-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:24.976289 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:24.976288    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx65l\" (UniqueName: \"kubernetes.io/projected/abd83ea1-a9ec-42f3-a1f1-2943ff79d809-kube-api-access-mx65l\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:25.103997 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:25.103922    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dc08ecb-c6d1-4db2-8170-ca4774a7d946" path="/var/lib/kubelet/pods/8dc08ecb-c6d1-4db2-8170-ca4774a7d946/volumes"
Nov 03 18:17:25.478975 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:25.478897    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2b4f46-jt8sd"]
Nov 03 18:17:25.479351 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:25.479338    5315 cpu_manager.go:414] "containerMap: removing container" podUID="abd83ea1-a9ec-42f3-a1f1-2943ff79d809" containerName="revoke-keys"
Nov 03 18:17:25.479403 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:25.479352    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abd83ea1-a9ec-42f3-a1f1-2943ff79d809" containerName="revoke-keys"
Nov 03 18:17:25.479446 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:25.479435    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="abd83ea1-a9ec-42f3-a1f1-2943ff79d809" containerName="revoke-keys"
Nov 03 18:17:25.482973 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:25.482806    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2b4f46-jt8sd"
Nov 03 18:17:25.543970 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:25.543935    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2b4f46-jt8sd"]
Nov 03 18:17:25.584249 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:25.584213    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7bca6e3-4297-4b19-9972-a6f444725f6e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-2b4f46-jt8sd\" (UID: \"d7bca6e3-4297-4b19-9972-a6f444725f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2b4f46-jt8sd"
Nov 03 18:17:25.584432 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:25.584403    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvtrj\" (UniqueName: \"kubernetes.io/projected/d7bca6e3-4297-4b19-9972-a6f444725f6e-kube-api-access-tvtrj\") pod \"maas-api-revoke-keys-e2e-worker-w5-2b4f46-jt8sd\" (UID: \"d7bca6e3-4297-4b19-9972-a6f444725f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2b4f46-jt8sd"
Nov 03 18:17:25.584513 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:25.584497    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7bca6e3-4297-4b19-9972-a6f444725f6e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-2b4f46-jt8sd\" (UID: \"d7bca6e3-4297-4b19-9972-a6f444725f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2b4f46-jt8sd"
Nov 03 18:17:25.653439 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:25.653410    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1720c00c-24tp8"
Nov 03 18:17:25.653439 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:25.653417    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1720c00c-24tp8" event={"ID":"abd83ea1-a9ec-42f3-a1f1-2943ff79d809","Type":"ContainerDied","Data":"2776d90a5037d9b0cdc3524e5bea89faa3f8e64eced66f566618854cc34e64d3"}
Nov 03 18:17:25.654021 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:25.653463    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2776d90a5037d9b0cdc3524e5bea89faa3f8e64eced66f566618854cc34e64d3"
Nov 03 18:17:25.685282 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:25.685250    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvtrj\" (UniqueName: \"kubernetes.io/projected/d7bca6e3-4297-4b19-9972-a6f444725f6e-kube-api-access-tvtrj\") pod \"maas-api-revoke-keys-e2e-worker-w5-2b4f46-jt8sd\" (UID: \"d7bca6e3-4297-4b19-9972-a6f444725f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2b4f46-jt8sd"
Nov 03 18:17:25.685459 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:25.685317    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7bca6e3-4297-4b19-9972-a6f444725f6e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-2b4f46-jt8sd\" (UID: \"d7bca6e3-4297-4b19-9972-a6f444725f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2b4f46-jt8sd"
Nov 03 18:17:25.685459 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:25.685337    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7bca6e3-4297-4b19-9972-a6f444725f6e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-2b4f46-jt8sd\" (UID: \"d7bca6e3-4297-4b19-9972-a6f444725f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2b4f46-jt8sd"
Nov 03 18:17:25.686157 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:25.686137    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7bca6e3-4297-4b19-9972-a6f444725f6e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-2b4f46-jt8sd\" (UID: \"d7bca6e3-4297-4b19-9972-a6f444725f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2b4f46-jt8sd"
Nov 03 18:17:25.687674 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:25.687655    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7bca6e3-4297-4b19-9972-a6f444725f6e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-2b4f46-jt8sd\" (UID: \"d7bca6e3-4297-4b19-9972-a6f444725f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2b4f46-jt8sd"
Nov 03 18:17:25.742510 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:25.742441    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvtrj\" (UniqueName: \"kubernetes.io/projected/d7bca6e3-4297-4b19-9972-a6f444725f6e-kube-api-access-tvtrj\") pod \"maas-api-revoke-keys-e2e-worker-w5-2b4f46-jt8sd\" (UID: \"d7bca6e3-4297-4b19-9972-a6f444725f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2b4f46-jt8sd"
Nov 03 18:17:25.796530 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:25.796493    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2b4f46-jt8sd"
Nov 03 18:17:26.167816 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:26.167781    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2b4f46-jt8sd"]
Nov 03 18:17:26.174483 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:17:26.174444    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7bca6e3_4297_4b19_9972_a6f444725f6e.slice/crio-dc6385c12dc0f07fcdd78d9573b4f7ec139ed5ac14c311c785b0da5ce1ccbc6d WatchSource:0}: Error finding container dc6385c12dc0f07fcdd78d9573b4f7ec139ed5ac14c311c785b0da5ce1ccbc6d: Status 404 returned error can't find the container with id dc6385c12dc0f07fcdd78d9573b4f7ec139ed5ac14c311c785b0da5ce1ccbc6d
Nov 03 18:17:26.661305 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:26.661271    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2b4f46-jt8sd" event={"ID":"d7bca6e3-4297-4b19-9972-a6f444725f6e","Type":"ContainerStarted","Data":"9af3796d4eaa197defae76ca6a355a21c41912f4120bd9008ab367439e851241"}
Nov 03 18:17:26.661753 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:26.661329    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2b4f46-jt8sd" event={"ID":"d7bca6e3-4297-4b19-9972-a6f444725f6e","Type":"ContainerStarted","Data":"dc6385c12dc0f07fcdd78d9573b4f7ec139ed5ac14c311c785b0da5ce1ccbc6d"}
Nov 03 18:17:26.770411 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:26.770277    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2b4f46-jt8sd" podStartSLOduration=1.770259629 podStartE2EDuration="1.770259629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:17:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:17:26.768701808 +0000 UTC m=+1885.784899877" watchObservedRunningTime="2026-11-03 18:17:26.770259629 +0000 UTC m=+1885.786457698"
Nov 03 18:17:27.669002 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:27.668968    5315 generic.go:356] "Generic (PLEG): container finished" podID="d7bca6e3-4297-4b19-9972-a6f444725f6e" containerID="9af3796d4eaa197defae76ca6a355a21c41912f4120bd9008ab367439e851241" exitCode=0
Nov 03 18:17:27.669382 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:27.669050    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2b4f46-jt8sd" event={"ID":"d7bca6e3-4297-4b19-9972-a6f444725f6e","Type":"ContainerDied","Data":"9af3796d4eaa197defae76ca6a355a21c41912f4120bd9008ab367439e851241"}
Nov 03 18:17:28.828334 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:28.828305    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2b4f46-jt8sd"
Nov 03 18:17:28.927657 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:28.927545    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7bca6e3-4297-4b19-9972-a6f444725f6e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7bca6e3-4297-4b19-9972-a6f444725f6e-maas-api-openshift-ca\") pod \"d7bca6e3-4297-4b19-9972-a6f444725f6e\" (UID: \"d7bca6e3-4297-4b19-9972-a6f444725f6e\") "
Nov 03 18:17:28.927657 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:28.927601    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7bca6e3-4297-4b19-9972-a6f444725f6e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7bca6e3-4297-4b19-9972-a6f444725f6e-maas-api-xks-ca\") pod \"d7bca6e3-4297-4b19-9972-a6f444725f6e\" (UID: \"d7bca6e3-4297-4b19-9972-a6f444725f6e\") "
Nov 03 18:17:28.927868 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:28.927677    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7bca6e3-4297-4b19-9972-a6f444725f6e-kube-api-access-tvtrj\" (UniqueName: \"kubernetes.io/projected/d7bca6e3-4297-4b19-9972-a6f444725f6e-kube-api-access-tvtrj\") pod \"d7bca6e3-4297-4b19-9972-a6f444725f6e\" (UID: \"d7bca6e3-4297-4b19-9972-a6f444725f6e\") "
Nov 03 18:17:28.927951 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:28.927926    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7bca6e3-4297-4b19-9972-a6f444725f6e-maas-api-openshift-ca" pod "d7bca6e3-4297-4b19-9972-a6f444725f6e" (UID: "d7bca6e3-4297-4b19-9972-a6f444725f6e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:17:28.928111 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:28.928092    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7bca6e3-4297-4b19-9972-a6f444725f6e-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:28.929888 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:28.929864    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7bca6e3-4297-4b19-9972-a6f444725f6e-kube-api-access-tvtrj" pod "d7bca6e3-4297-4b19-9972-a6f444725f6e" (UID: "d7bca6e3-4297-4b19-9972-a6f444725f6e"). InnerVolumeSpecName "kube-api-access-tvtrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:17:28.930170 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:28.930152    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7bca6e3-4297-4b19-9972-a6f444725f6e-maas-api-xks-ca" pod "d7bca6e3-4297-4b19-9972-a6f444725f6e" (UID: "d7bca6e3-4297-4b19-9972-a6f444725f6e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:17:29.029736 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:29.029692    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvtrj\" (UniqueName: \"kubernetes.io/projected/d7bca6e3-4297-4b19-9972-a6f444725f6e-kube-api-access-tvtrj\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:29.029736 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:29.029731    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7bca6e3-4297-4b19-9972-a6f444725f6e-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:29.683959 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:29.683930    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2b4f46-jt8sd"
Nov 03 18:17:29.684151 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:29.683961    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2b4f46-jt8sd" event={"ID":"d7bca6e3-4297-4b19-9972-a6f444725f6e","Type":"ContainerDied","Data":"dc6385c12dc0f07fcdd78d9573b4f7ec139ed5ac14c311c785b0da5ce1ccbc6d"}
Nov 03 18:17:29.684151 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:29.684023    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc6385c12dc0f07fcdd78d9573b4f7ec139ed5ac14c311c785b0da5ce1ccbc6d"
Nov 03 18:17:29.686557 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:29.686520    5315 generic.go:356] "Generic (PLEG): container finished" podID="915e71a9-f7c9-4be4-929a-af6be579b6bd" containerID="8f4337e2ad655e0367f10693fec2249a1c5648782bc150d4bae14cad4d970281" exitCode=0
Nov 03 18:17:29.686747 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:29.686554    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g" event={"ID":"915e71a9-f7c9-4be4-929a-af6be579b6bd","Type":"ContainerDied","Data":"8f4337e2ad655e0367f10693fec2249a1c5648782bc150d4bae14cad4d970281"}
Nov 03 18:17:29.686747 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:29.686605    5315 scope.go:122] "RemoveContainer" containerID="8075f30f685a50373bf0aad4c0e226ba8e9fae327819f934e88dc058ed053ce1"
Nov 03 18:17:29.724259 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:29.724234    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g"
Nov 03 18:17:29.837896 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:29.837869    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/915e71a9-f7c9-4be4-929a-af6be579b6bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/915e71a9-f7c9-4be4-929a-af6be579b6bd-istiod-ca-cert\") pod \"915e71a9-f7c9-4be4-929a-af6be579b6bd\" (UID: \"915e71a9-f7c9-4be4-929a-af6be579b6bd\") "
Nov 03 18:17:29.838377 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:29.837922    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/915e71a9-f7c9-4be4-929a-af6be579b6bd-istio-token\" (UniqueName: \"kubernetes.io/projected/915e71a9-f7c9-4be4-929a-af6be579b6bd-istio-token\") pod \"915e71a9-f7c9-4be4-929a-af6be579b6bd\" (UID: \"915e71a9-f7c9-4be4-929a-af6be579b6bd\") "
Nov 03 18:17:29.838377 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:29.837956    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/915e71a9-f7c9-4be4-929a-af6be579b6bd-kube-api-access-7zxcd\" (UniqueName: \"kubernetes.io/projected/915e71a9-f7c9-4be4-929a-af6be579b6bd-kube-api-access-7zxcd\") pod \"915e71a9-f7c9-4be4-929a-af6be579b6bd\" (UID: \"915e71a9-f7c9-4be4-929a-af6be579b6bd\") "
Nov 03 18:17:29.838377 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:29.838004    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/915e71a9-f7c9-4be4-929a-af6be579b6bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/915e71a9-f7c9-4be4-929a-af6be579b6bd-istio-envoy\") pod \"915e71a9-f7c9-4be4-929a-af6be579b6bd\" (UID: \"915e71a9-f7c9-4be4-929a-af6be579b6bd\") "
Nov 03 18:17:29.838377 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:29.838067    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/915e71a9-f7c9-4be4-929a-af6be579b6bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/915e71a9-f7c9-4be4-929a-af6be579b6bd-workload-socket\") pod \"915e71a9-f7c9-4be4-929a-af6be579b6bd\" (UID: \"915e71a9-f7c9-4be4-929a-af6be579b6bd\") "
Nov 03 18:17:29.838377 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:29.838100    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/915e71a9-f7c9-4be4-929a-af6be579b6bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/915e71a9-f7c9-4be4-929a-af6be579b6bd-istio-podinfo\") pod \"915e71a9-f7c9-4be4-929a-af6be579b6bd\" (UID: \"915e71a9-f7c9-4be4-929a-af6be579b6bd\") "
Nov 03 18:17:29.838377 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:29.838149    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/915e71a9-f7c9-4be4-929a-af6be579b6bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/915e71a9-f7c9-4be4-929a-af6be579b6bd-credential-socket\") pod \"915e71a9-f7c9-4be4-929a-af6be579b6bd\" (UID: \"915e71a9-f7c9-4be4-929a-af6be579b6bd\") "
Nov 03 18:17:29.838377 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:29.838175    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/915e71a9-f7c9-4be4-929a-af6be579b6bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/915e71a9-f7c9-4be4-929a-af6be579b6bd-workload-certs\") pod \"915e71a9-f7c9-4be4-929a-af6be579b6bd\" (UID: \"915e71a9-f7c9-4be4-929a-af6be579b6bd\") "
Nov 03 18:17:29.838377 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:29.838251    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/915e71a9-f7c9-4be4-929a-af6be579b6bd-istiod-ca-cert" pod "915e71a9-f7c9-4be4-929a-af6be579b6bd" (UID: "915e71a9-f7c9-4be4-929a-af6be579b6bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:17:29.838377 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:29.838263    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/915e71a9-f7c9-4be4-929a-af6be579b6bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/915e71a9-f7c9-4be4-929a-af6be579b6bd-istio-data\") pod \"915e71a9-f7c9-4be4-929a-af6be579b6bd\" (UID: \"915e71a9-f7c9-4be4-929a-af6be579b6bd\") "
Nov 03 18:17:29.838886 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:29.838553    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/915e71a9-f7c9-4be4-929a-af6be579b6bd-workload-socket" pod "915e71a9-f7c9-4be4-929a-af6be579b6bd" (UID: "915e71a9-f7c9-4be4-929a-af6be579b6bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:29.838886 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:29.838692    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/915e71a9-f7c9-4be4-929a-af6be579b6bd-workload-certs" pod "915e71a9-f7c9-4be4-929a-af6be579b6bd" (UID: "915e71a9-f7c9-4be4-929a-af6be579b6bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:29.838886 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:29.838782    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/915e71a9-f7c9-4be4-929a-af6be579b6bd-istio-data" pod "915e71a9-f7c9-4be4-929a-af6be579b6bd" (UID: "915e71a9-f7c9-4be4-929a-af6be579b6bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:29.838886 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:29.838802    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/915e71a9-f7c9-4be4-929a-af6be579b6bd-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:29.838886 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:29.838826    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/915e71a9-f7c9-4be4-929a-af6be579b6bd-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:29.839251 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:29.838900    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/915e71a9-f7c9-4be4-929a-af6be579b6bd-credential-socket" pod "915e71a9-f7c9-4be4-929a-af6be579b6bd" (UID: "915e71a9-f7c9-4be4-929a-af6be579b6bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:29.840495 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:29.840470    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/915e71a9-f7c9-4be4-929a-af6be579b6bd-istio-token" pod "915e71a9-f7c9-4be4-929a-af6be579b6bd" (UID: "915e71a9-f7c9-4be4-929a-af6be579b6bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:17:29.840650 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:29.840626    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/915e71a9-f7c9-4be4-929a-af6be579b6bd-istio-podinfo" pod "915e71a9-f7c9-4be4-929a-af6be579b6bd" (UID: "915e71a9-f7c9-4be4-929a-af6be579b6bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:17:29.840851 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:29.840829    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/915e71a9-f7c9-4be4-929a-af6be579b6bd-kube-api-access-7zxcd" pod "915e71a9-f7c9-4be4-929a-af6be579b6bd" (UID: "915e71a9-f7c9-4be4-929a-af6be579b6bd"). InnerVolumeSpecName "kube-api-access-7zxcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:17:29.840899 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:29.840853    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/915e71a9-f7c9-4be4-929a-af6be579b6bd-istio-envoy" pod "915e71a9-f7c9-4be4-929a-af6be579b6bd" (UID: "915e71a9-f7c9-4be4-929a-af6be579b6bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:29.940071 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:29.940030    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/915e71a9-f7c9-4be4-929a-af6be579b6bd-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:29.940071 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:29.940065    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zxcd\" (UniqueName: \"kubernetes.io/projected/915e71a9-f7c9-4be4-929a-af6be579b6bd-kube-api-access-7zxcd\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:29.940284 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:29.940081    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/915e71a9-f7c9-4be4-929a-af6be579b6bd-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:29.940284 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:29.940090    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/915e71a9-f7c9-4be4-929a-af6be579b6bd-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:29.940284 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:29.940099    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/915e71a9-f7c9-4be4-929a-af6be579b6bd-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:29.940284 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:29.940109    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/915e71a9-f7c9-4be4-929a-af6be579b6bd-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:29.940284 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:29.940117    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/915e71a9-f7c9-4be4-929a-af6be579b6bd-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:29.961480 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:29.961445    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr"]
Nov 03 18:17:29.985696 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:29.985664    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l"]
Nov 03 18:17:30.694026 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:30.693987    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g"
Nov 03 18:17:30.694246 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:30.694032    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g" event={"ID":"915e71a9-f7c9-4be4-929a-af6be579b6bd","Type":"ContainerDied","Data":"7e936087cb388bc5625bd53ab6d23b4a86ab6db0da6974a7015d62aeed2e87ff"}
Nov 03 18:17:30.694246 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:30.694105    5315 scope.go:122] "RemoveContainer" containerID="8f4337e2ad655e0367f10693fec2249a1c5648782bc150d4bae14cad4d970281"
Nov 03 18:17:30.858670 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:30.858133    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g"]
Nov 03 18:17:30.896666 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:30.896632    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-232f9e-openshift-default-c868855cc-77n5g"]
Nov 03 18:17:31.103241 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:31.103197    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="915e71a9-f7c9-4be4-929a-af6be579b6bd" path="/var/lib/kubelet/pods/915e71a9-f7c9-4be4-929a-af6be579b6bd/volumes"
Nov 03 18:17:32.263456 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:32.263407    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f79c02-b4bk8"]
Nov 03 18:17:32.309392 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:32.309357    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f79c02-b4bk8"]
Nov 03 18:17:33.107260 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:33.107206    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c977dc59-b70d-4cc0-9531-d76c172d7b52" path="/var/lib/kubelet/pods/c977dc59-b70d-4cc0-9531-d76c172d7b52/volumes"
Nov 03 18:17:33.771230 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:33.771192    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1720c00c-7f49c64bcf-dnv6v"]
Nov 03 18:17:33.890357 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:33.890239    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1720c00c-549d8c448c-lzrvl"]
Nov 03 18:17:36.327217 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:36.327182    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b04d452e-wlv5h"]
Nov 03 18:17:36.327653 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:36.327627    5315 cpu_manager.go:414] "containerMap: removing container" podUID="915e71a9-f7c9-4be4-929a-af6be579b6bd" containerName="istio-proxy"
Nov 03 18:17:36.327653 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:36.327639    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="915e71a9-f7c9-4be4-929a-af6be579b6bd" containerName="istio-proxy"
Nov 03 18:17:36.327730 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:36.327664    5315 cpu_manager.go:414] "containerMap: removing container" podUID="915e71a9-f7c9-4be4-929a-af6be579b6bd" containerName="istio-proxy"
Nov 03 18:17:36.327730 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:36.327671    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="915e71a9-f7c9-4be4-929a-af6be579b6bd" containerName="istio-proxy"
Nov 03 18:17:36.327730 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:36.327690    5315 cpu_manager.go:414] "containerMap: removing container" podUID="d7bca6e3-4297-4b19-9972-a6f444725f6e" containerName="revoke-keys"
Nov 03 18:17:36.327730 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:36.327695    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7bca6e3-4297-4b19-9972-a6f444725f6e" containerName="revoke-keys"
Nov 03 18:17:36.327900 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:36.327762    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="915e71a9-f7c9-4be4-929a-af6be579b6bd" containerName="istio-proxy"
Nov 03 18:17:36.327900 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:36.327775    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="915e71a9-f7c9-4be4-929a-af6be579b6bd" containerName="istio-proxy"
Nov 03 18:17:36.327900 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:36.327785    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7bca6e3-4297-4b19-9972-a6f444725f6e" containerName="revoke-keys"
Nov 03 18:17:36.332680 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:36.332650    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8941c20-8kr5x"]
Nov 03 18:17:36.332973 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:36.332953    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04d452e-wlv5h"
Nov 03 18:17:36.337274 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:36.337246    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8941c20-8kr5x"
Nov 03 18:17:36.381449 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:36.381415    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:17:36.381449 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:36.381442    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:17:36.384090 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:36.384043    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:17:36.431577 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:36.431539    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b04d452e-wlv5h"]
Nov 03 18:17:36.435221 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:36.435194    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8941c20-8kr5x"]
Nov 03 18:17:36.454477 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:36.454426    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e376e147-b415-47d0-8bb8-21aecf441da9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b04d452e-wlv5h\" (UID: \"e376e147-b415-47d0-8bb8-21aecf441da9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04d452e-wlv5h"
Nov 03 18:17:36.454713 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:36.454559    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrrhh\" (UniqueName: \"kubernetes.io/projected/91b95076-e971-49d4-b183-66e0da518d56-kube-api-access-nrrhh\") pod \"maas-api-revoke-sub-c8941c20-8kr5x\" (UID: \"91b95076-e971-49d4-b183-66e0da518d56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8941c20-8kr5x"
Nov 03 18:17:36.454713 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:36.454641    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ngk2\" (UniqueName: \"kubernetes.io/projected/e376e147-b415-47d0-8bb8-21aecf441da9-kube-api-access-9ngk2\") pod \"maas-api-revoke-sub-b04d452e-wlv5h\" (UID: \"e376e147-b415-47d0-8bb8-21aecf441da9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04d452e-wlv5h"
Nov 03 18:17:36.454855 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:36.454709    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91b95076-e971-49d4-b183-66e0da518d56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8941c20-8kr5x\" (UID: \"91b95076-e971-49d4-b183-66e0da518d56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8941c20-8kr5x"
Nov 03 18:17:36.454855 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:36.454780    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e376e147-b415-47d0-8bb8-21aecf441da9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b04d452e-wlv5h\" (UID: \"e376e147-b415-47d0-8bb8-21aecf441da9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04d452e-wlv5h"
Nov 03 18:17:36.454984 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:36.454905    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91b95076-e971-49d4-b183-66e0da518d56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8941c20-8kr5x\" (UID: \"91b95076-e971-49d4-b183-66e0da518d56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8941c20-8kr5x"
Nov 03 18:17:36.481638 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:36.481579    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:17:36.556024 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:36.555982    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e376e147-b415-47d0-8bb8-21aecf441da9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b04d452e-wlv5h\" (UID: \"e376e147-b415-47d0-8bb8-21aecf441da9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04d452e-wlv5h"
Nov 03 18:17:36.556239 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:36.556060    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91b95076-e971-49d4-b183-66e0da518d56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8941c20-8kr5x\" (UID: \"91b95076-e971-49d4-b183-66e0da518d56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8941c20-8kr5x"
Nov 03 18:17:36.556239 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:36.556099    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e376e147-b415-47d0-8bb8-21aecf441da9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b04d452e-wlv5h\" (UID: \"e376e147-b415-47d0-8bb8-21aecf441da9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04d452e-wlv5h"
Nov 03 18:17:36.556239 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:36.556224    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrrhh\" (UniqueName: \"kubernetes.io/projected/91b95076-e971-49d4-b183-66e0da518d56-kube-api-access-nrrhh\") pod \"maas-api-revoke-sub-c8941c20-8kr5x\" (UID: \"91b95076-e971-49d4-b183-66e0da518d56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8941c20-8kr5x"
Nov 03 18:17:36.556444 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:36.556275    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ngk2\" (UniqueName: \"kubernetes.io/projected/e376e147-b415-47d0-8bb8-21aecf441da9-kube-api-access-9ngk2\") pod \"maas-api-revoke-sub-b04d452e-wlv5h\" (UID: \"e376e147-b415-47d0-8bb8-21aecf441da9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04d452e-wlv5h"
Nov 03 18:17:36.556444 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:36.556316    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91b95076-e971-49d4-b183-66e0da518d56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8941c20-8kr5x\" (UID: \"91b95076-e971-49d4-b183-66e0da518d56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8941c20-8kr5x"
Nov 03 18:17:36.556899 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:36.556870    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91b95076-e971-49d4-b183-66e0da518d56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8941c20-8kr5x\" (UID: \"91b95076-e971-49d4-b183-66e0da518d56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8941c20-8kr5x"
Nov 03 18:17:36.557014 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:36.556874    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e376e147-b415-47d0-8bb8-21aecf441da9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b04d452e-wlv5h\" (UID: \"e376e147-b415-47d0-8bb8-21aecf441da9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04d452e-wlv5h"
Nov 03 18:17:36.558469 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:36.558447    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e376e147-b415-47d0-8bb8-21aecf441da9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b04d452e-wlv5h\" (UID: \"e376e147-b415-47d0-8bb8-21aecf441da9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04d452e-wlv5h"
Nov 03 18:17:36.558573 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:36.558556    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91b95076-e971-49d4-b183-66e0da518d56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8941c20-8kr5x\" (UID: \"91b95076-e971-49d4-b183-66e0da518d56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8941c20-8kr5x"
Nov 03 18:17:36.613593 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:36.613516    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrrhh\" (UniqueName: \"kubernetes.io/projected/91b95076-e971-49d4-b183-66e0da518d56-kube-api-access-nrrhh\") pod \"maas-api-revoke-sub-c8941c20-8kr5x\" (UID: \"91b95076-e971-49d4-b183-66e0da518d56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8941c20-8kr5x"
Nov 03 18:17:36.613593 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:36.613568    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ngk2\" (UniqueName: \"kubernetes.io/projected/e376e147-b415-47d0-8bb8-21aecf441da9-kube-api-access-9ngk2\") pod \"maas-api-revoke-sub-b04d452e-wlv5h\" (UID: \"e376e147-b415-47d0-8bb8-21aecf441da9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04d452e-wlv5h"
Nov 03 18:17:36.648565 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:36.648534    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04d452e-wlv5h"
Nov 03 18:17:36.654668 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:36.654640    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8941c20-8kr5x"
Nov 03 18:17:37.113245 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:17:37.113205    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode376e147_b415_47d0_8bb8_21aecf441da9.slice/crio-47467a4278e916f71b319916e0eaf29699a676d6d4403e95bfae3b806ae84262 WatchSource:0}: Error finding container 47467a4278e916f71b319916e0eaf29699a676d6d4403e95bfae3b806ae84262: Status 404 returned error can't find the container with id 47467a4278e916f71b319916e0eaf29699a676d6d4403e95bfae3b806ae84262
Nov 03 18:17:37.116254 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:37.116221    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b04d452e-wlv5h"]
Nov 03 18:17:37.138302 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:37.138267    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8941c20-8kr5x"]
Nov 03 18:17:37.145900 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:17:37.145868    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91b95076_e971_49d4_b183_66e0da518d56.slice/crio-714c7b7811161f9a3a653d3d30642c2a9b09e686483e0cf03603bd06eed86085 WatchSource:0}: Error finding container 714c7b7811161f9a3a653d3d30642c2a9b09e686483e0cf03603bd06eed86085: Status 404 returned error can't find the container with id 714c7b7811161f9a3a653d3d30642c2a9b09e686483e0cf03603bd06eed86085
Nov 03 18:17:37.786712 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:37.785770    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8941c20-8kr5x" event={"ID":"91b95076-e971-49d4-b183-66e0da518d56","Type":"ContainerStarted","Data":"d4d0a252baff91c18f15dd0a83dfdf8fcf5d66564446ab519cca916a8fb5b0a6"}
Nov 03 18:17:37.786712 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:37.785842    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8941c20-8kr5x" event={"ID":"91b95076-e971-49d4-b183-66e0da518d56","Type":"ContainerStarted","Data":"714c7b7811161f9a3a653d3d30642c2a9b09e686483e0cf03603bd06eed86085"}
Nov 03 18:17:37.791998 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:37.791062    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04d452e-wlv5h" event={"ID":"e376e147-b415-47d0-8bb8-21aecf441da9","Type":"ContainerStarted","Data":"8c74fbe22f327b2ee29d5d0c6ab5186e25693ac6516fe5ad69299c54e0abbbaf"}
Nov 03 18:17:37.791998 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:37.791127    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04d452e-wlv5h" event={"ID":"e376e147-b415-47d0-8bb8-21aecf441da9","Type":"ContainerStarted","Data":"47467a4278e916f71b319916e0eaf29699a676d6d4403e95bfae3b806ae84262"}
Nov 03 18:17:37.897112 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:37.897054    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8941c20-8kr5x" podStartSLOduration=1.8970409240000001 podStartE2EDuration="1.897040924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:17:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:17:37.895527627 +0000 UTC m=+1896.911725688" watchObservedRunningTime="2026-11-03 18:17:37.897040924 +0000 UTC m=+1896.913239020"
Nov 03 18:17:38.025529 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:38.025465    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04d452e-wlv5h" podStartSLOduration=2.025448194 podStartE2EDuration="2.025448194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:17:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:17:38.022985255 +0000 UTC m=+1897.039183325" watchObservedRunningTime="2026-11-03 18:17:38.025448194 +0000 UTC m=+1897.041646262"
Nov 03 18:17:38.771755 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:38.771681    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-1720c00c-7f49c64bcf-dnv6v" podUID="2c20ab79-c080-49c3-a84e-f8f3fc0713f2" containerName="payload-processing" containerID="cri-o://439eb85ffd330e0f188e970555768dd2944b1512da3f7852af9f1d779fb5408a" gracePeriod=25
Nov 03 18:17:38.798554 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:38.798521    5315 generic.go:356] "Generic (PLEG): container finished" podID="e376e147-b415-47d0-8bb8-21aecf441da9" containerID="8c74fbe22f327b2ee29d5d0c6ab5186e25693ac6516fe5ad69299c54e0abbbaf" exitCode=0
Nov 03 18:17:38.799038 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:38.798605    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04d452e-wlv5h" event={"ID":"e376e147-b415-47d0-8bb8-21aecf441da9","Type":"ContainerDied","Data":"8c74fbe22f327b2ee29d5d0c6ab5186e25693ac6516fe5ad69299c54e0abbbaf"}
Nov 03 18:17:38.800409 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:38.800386    5315 generic.go:356] "Generic (PLEG): container finished" podID="91b95076-e971-49d4-b183-66e0da518d56" containerID="d4d0a252baff91c18f15dd0a83dfdf8fcf5d66564446ab519cca916a8fb5b0a6" exitCode=0
Nov 03 18:17:38.800520 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:38.800472    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8941c20-8kr5x" event={"ID":"91b95076-e971-49d4-b183-66e0da518d56","Type":"ContainerDied","Data":"d4d0a252baff91c18f15dd0a83dfdf8fcf5d66564446ab519cca916a8fb5b0a6"}
Nov 03 18:17:38.891189 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:38.891146    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-1720c00c-549d8c448c-lzrvl" podUID="fc4effb6-ba04-4eab-aeaf-a80ef45d2acb" containerName="payload-pre-processing" containerID="cri-o://2f7089861bfcff1ed9a705ed7d9c7d4f15071ddaa018022f4ea24c181ed3b10c" gracePeriod=25
Nov 03 18:17:39.027359 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:39.027326    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1720c00c-7f49c64bcf-dnv6v"
Nov 03 18:17:39.107956 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:39.107920    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c20ab79-c080-49c3-a84e-f8f3fc0713f2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c20ab79-c080-49c3-a84e-f8f3fc0713f2-plugins-config-volume\") pod \"2c20ab79-c080-49c3-a84e-f8f3fc0713f2\" (UID: \"2c20ab79-c080-49c3-a84e-f8f3fc0713f2\") "
Nov 03 18:17:39.108227 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:39.108202    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c20ab79-c080-49c3-a84e-f8f3fc0713f2-kube-api-access-skv7q\" (UniqueName: \"kubernetes.io/projected/2c20ab79-c080-49c3-a84e-f8f3fc0713f2-kube-api-access-skv7q\") pod \"2c20ab79-c080-49c3-a84e-f8f3fc0713f2\" (UID: \"2c20ab79-c080-49c3-a84e-f8f3fc0713f2\") "
Nov 03 18:17:39.108369 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:39.108345    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c20ab79-c080-49c3-a84e-f8f3fc0713f2-plugins-config-volume" pod "2c20ab79-c080-49c3-a84e-f8f3fc0713f2" (UID: "2c20ab79-c080-49c3-a84e-f8f3fc0713f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:17:39.108762 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:39.108730    5315 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c20ab79-c080-49c3-a84e-f8f3fc0713f2-plugins-config-volume\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:39.111651 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:39.110543    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c20ab79-c080-49c3-a84e-f8f3fc0713f2-kube-api-access-skv7q" pod "2c20ab79-c080-49c3-a84e-f8f3fc0713f2" (UID: "2c20ab79-c080-49c3-a84e-f8f3fc0713f2"). InnerVolumeSpecName "kube-api-access-skv7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:17:39.156881 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:39.156852    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1720c00c-549d8c448c-lzrvl"
Nov 03 18:17:39.210505 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:39.210467    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skv7q\" (UniqueName: \"kubernetes.io/projected/2c20ab79-c080-49c3-a84e-f8f3fc0713f2-kube-api-access-skv7q\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:39.311949 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:39.311869    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc4effb6-ba04-4eab-aeaf-a80ef45d2acb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc4effb6-ba04-4eab-aeaf-a80ef45d2acb-plugins-config-volume\") pod \"fc4effb6-ba04-4eab-aeaf-a80ef45d2acb\" (UID: \"fc4effb6-ba04-4eab-aeaf-a80ef45d2acb\") "
Nov 03 18:17:39.312127 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:39.312002    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc4effb6-ba04-4eab-aeaf-a80ef45d2acb-kube-api-access-np78v\" (UniqueName: \"kubernetes.io/projected/fc4effb6-ba04-4eab-aeaf-a80ef45d2acb-kube-api-access-np78v\") pod \"fc4effb6-ba04-4eab-aeaf-a80ef45d2acb\" (UID: \"fc4effb6-ba04-4eab-aeaf-a80ef45d2acb\") "
Nov 03 18:17:39.312325 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:39.312300    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc4effb6-ba04-4eab-aeaf-a80ef45d2acb-plugins-config-volume" pod "fc4effb6-ba04-4eab-aeaf-a80ef45d2acb" (UID: "fc4effb6-ba04-4eab-aeaf-a80ef45d2acb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:17:39.314122 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:39.314099    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc4effb6-ba04-4eab-aeaf-a80ef45d2acb-kube-api-access-np78v" pod "fc4effb6-ba04-4eab-aeaf-a80ef45d2acb" (UID: "fc4effb6-ba04-4eab-aeaf-a80ef45d2acb"). InnerVolumeSpecName "kube-api-access-np78v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:17:39.413674 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:39.413637    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np78v\" (UniqueName: \"kubernetes.io/projected/fc4effb6-ba04-4eab-aeaf-a80ef45d2acb-kube-api-access-np78v\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:39.413674 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:39.413671    5315 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc4effb6-ba04-4eab-aeaf-a80ef45d2acb-plugins-config-volume\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:39.727871 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:39.727774    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn"]
Nov 03 18:17:39.728106 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:39.728066    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn" podUID="cb78e2af-5c83-4d6d-a562-274b12209c20" containerName="istio-proxy" containerID="cri-o://6da1959a991a4e6d878dc7df2369f8a6a53f12464d9442d98ed7737ca80b9eef" gracePeriod=30
Nov 03 18:17:39.730204 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:39.730177    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn"]
Nov 03 18:17:39.730364 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:39.730345    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn" podUID="dafbc7c7-538e-4bf6-ab30-970870f21854" containerName="istio-proxy" containerID="cri-o://131139a80d6fb7ad51d73854a2036d32c7d660b479bc2c84808e89e370b8592a" gracePeriod=30
Nov 03 18:17:39.815540 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:39.815494    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1720c00c-549d8c448c-lzrvl" event={"ID":"fc4effb6-ba04-4eab-aeaf-a80ef45d2acb","Type":"ContainerDied","Data":"2f7089861bfcff1ed9a705ed7d9c7d4f15071ddaa018022f4ea24c181ed3b10c"}
Nov 03 18:17:39.816060 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:39.815534    5315 generic.go:356] "Generic (PLEG): container finished" podID="fc4effb6-ba04-4eab-aeaf-a80ef45d2acb" containerID="2f7089861bfcff1ed9a705ed7d9c7d4f15071ddaa018022f4ea24c181ed3b10c" exitCode=0
Nov 03 18:17:39.816060 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:39.815591    5315 scope.go:122] "RemoveContainer" containerID="2f7089861bfcff1ed9a705ed7d9c7d4f15071ddaa018022f4ea24c181ed3b10c"
Nov 03 18:17:39.816060 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:39.815692    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1720c00c-549d8c448c-lzrvl" event={"ID":"fc4effb6-ba04-4eab-aeaf-a80ef45d2acb","Type":"ContainerDied","Data":"124db5dc6265f8960896e94f938160f510f5aa55de42b8020bc5d0ed0f82d405"}
Nov 03 18:17:39.816060 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:39.815826    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1720c00c-549d8c448c-lzrvl"
Nov 03 18:17:39.821185 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:39.819361    5315 generic.go:356] "Generic (PLEG): container finished" podID="2c20ab79-c080-49c3-a84e-f8f3fc0713f2" containerID="439eb85ffd330e0f188e970555768dd2944b1512da3f7852af9f1d779fb5408a" exitCode=0
Nov 03 18:17:39.821185 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:39.820392    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1720c00c-7f49c64bcf-dnv6v"
Nov 03 18:17:39.864679 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:39.855541    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1720c00c-7f49c64bcf-dnv6v" event={"ID":"2c20ab79-c080-49c3-a84e-f8f3fc0713f2","Type":"ContainerDied","Data":"439eb85ffd330e0f188e970555768dd2944b1512da3f7852af9f1d779fb5408a"}
Nov 03 18:17:39.864679 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:39.855641    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1720c00c-7f49c64bcf-dnv6v" event={"ID":"2c20ab79-c080-49c3-a84e-f8f3fc0713f2","Type":"ContainerDied","Data":"88a5df0fa8bbb5e3593e7cf5c514395972f532cb451f95eca4666e5fbd59fea6"}
Nov 03 18:17:39.930157 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:39.929999    5315 scope.go:122] "RemoveContainer" containerID="2f7089861bfcff1ed9a705ed7d9c7d4f15071ddaa018022f4ea24c181ed3b10c"
Nov 03 18:17:39.930660 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:17:39.930421    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f7089861bfcff1ed9a705ed7d9c7d4f15071ddaa018022f4ea24c181ed3b10c\": container with ID starting with 2f7089861bfcff1ed9a705ed7d9c7d4f15071ddaa018022f4ea24c181ed3b10c not found: ID does not exist" containerID="2f7089861bfcff1ed9a705ed7d9c7d4f15071ddaa018022f4ea24c181ed3b10c"
Nov 03 18:17:39.930660 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:39.930462    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f7089861bfcff1ed9a705ed7d9c7d4f15071ddaa018022f4ea24c181ed3b10c"} err="failed to get container status \"2f7089861bfcff1ed9a705ed7d9c7d4f15071ddaa018022f4ea24c181ed3b10c\": rpc error: code = NotFound desc = could not find container \"2f7089861bfcff1ed9a705ed7d9c7d4f15071ddaa018022f4ea24c181ed3b10c\": container with ID starting with 2f7089861bfcff1ed9a705ed7d9c7d4f15071ddaa018022f4ea24c181ed3b10c not found: ID does not exist"
Nov 03 18:17:39.930660 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:39.930487    5315 scope.go:122] "RemoveContainer" containerID="439eb85ffd330e0f188e970555768dd2944b1512da3f7852af9f1d779fb5408a"
Nov 03 18:17:39.943733 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:39.943712    5315 scope.go:122] "RemoveContainer" containerID="439eb85ffd330e0f188e970555768dd2944b1512da3f7852af9f1d779fb5408a"
Nov 03 18:17:39.944097 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:17:39.944073    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"439eb85ffd330e0f188e970555768dd2944b1512da3f7852af9f1d779fb5408a\": container with ID starting with 439eb85ffd330e0f188e970555768dd2944b1512da3f7852af9f1d779fb5408a not found: ID does not exist" containerID="439eb85ffd330e0f188e970555768dd2944b1512da3f7852af9f1d779fb5408a"
Nov 03 18:17:39.944182 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:39.944110    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"439eb85ffd330e0f188e970555768dd2944b1512da3f7852af9f1d779fb5408a"} err="failed to get container status \"439eb85ffd330e0f188e970555768dd2944b1512da3f7852af9f1d779fb5408a\": rpc error: code = NotFound desc = could not find container \"439eb85ffd330e0f188e970555768dd2944b1512da3f7852af9f1d779fb5408a\": container with ID starting with 439eb85ffd330e0f188e970555768dd2944b1512da3f7852af9f1d779fb5408a not found: ID does not exist"
Nov 03 18:17:40.064724 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:40.064694    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04d452e-wlv5h"
Nov 03 18:17:40.083943 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:40.083917    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8941c20-8kr5x"
Nov 03 18:17:40.172163 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:40.172126    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1720c00c-549d8c448c-lzrvl"]
Nov 03 18:17:40.177109 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:40.177081    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1720c00c-549d8c448c-lzrvl"]
Nov 03 18:17:40.179862 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:40.179835    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91b95076-e971-49d4-b183-66e0da518d56-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91b95076-e971-49d4-b183-66e0da518d56-maas-api-xks-ca\") pod \"91b95076-e971-49d4-b183-66e0da518d56\" (UID: \"91b95076-e971-49d4-b183-66e0da518d56\") "
Nov 03 18:17:40.180001 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:40.179911    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91b95076-e971-49d4-b183-66e0da518d56-kube-api-access-nrrhh\" (UniqueName: \"kubernetes.io/projected/91b95076-e971-49d4-b183-66e0da518d56-kube-api-access-nrrhh\") pod \"91b95076-e971-49d4-b183-66e0da518d56\" (UID: \"91b95076-e971-49d4-b183-66e0da518d56\") "
Nov 03 18:17:40.180001 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:40.179973    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e376e147-b415-47d0-8bb8-21aecf441da9-kube-api-access-9ngk2\" (UniqueName: \"kubernetes.io/projected/e376e147-b415-47d0-8bb8-21aecf441da9-kube-api-access-9ngk2\") pod \"e376e147-b415-47d0-8bb8-21aecf441da9\" (UID: \"e376e147-b415-47d0-8bb8-21aecf441da9\") "
Nov 03 18:17:40.180094 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:40.180005    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e376e147-b415-47d0-8bb8-21aecf441da9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e376e147-b415-47d0-8bb8-21aecf441da9-maas-api-xks-ca\") pod \"e376e147-b415-47d0-8bb8-21aecf441da9\" (UID: \"e376e147-b415-47d0-8bb8-21aecf441da9\") "
Nov 03 18:17:40.180094 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:40.180041    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e376e147-b415-47d0-8bb8-21aecf441da9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e376e147-b415-47d0-8bb8-21aecf441da9-maas-api-openshift-ca\") pod \"e376e147-b415-47d0-8bb8-21aecf441da9\" (UID: \"e376e147-b415-47d0-8bb8-21aecf441da9\") "
Nov 03 18:17:40.180094 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:40.180080    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91b95076-e971-49d4-b183-66e0da518d56-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91b95076-e971-49d4-b183-66e0da518d56-maas-api-openshift-ca\") pod \"91b95076-e971-49d4-b183-66e0da518d56\" (UID: \"91b95076-e971-49d4-b183-66e0da518d56\") "
Nov 03 18:17:40.180473 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:40.180421    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e376e147-b415-47d0-8bb8-21aecf441da9-maas-api-openshift-ca" pod "e376e147-b415-47d0-8bb8-21aecf441da9" (UID: "e376e147-b415-47d0-8bb8-21aecf441da9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:17:40.180603 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:40.180484    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91b95076-e971-49d4-b183-66e0da518d56-maas-api-openshift-ca" pod "91b95076-e971-49d4-b183-66e0da518d56" (UID: "91b95076-e971-49d4-b183-66e0da518d56"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:17:40.182291 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:40.182268    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e376e147-b415-47d0-8bb8-21aecf441da9-kube-api-access-9ngk2" pod "e376e147-b415-47d0-8bb8-21aecf441da9" (UID: "e376e147-b415-47d0-8bb8-21aecf441da9"). InnerVolumeSpecName "kube-api-access-9ngk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:17:40.182427 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:40.182372    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e376e147-b415-47d0-8bb8-21aecf441da9-maas-api-xks-ca" pod "e376e147-b415-47d0-8bb8-21aecf441da9" (UID: "e376e147-b415-47d0-8bb8-21aecf441da9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:17:40.182494 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:40.182477    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91b95076-e971-49d4-b183-66e0da518d56-maas-api-xks-ca" pod "91b95076-e971-49d4-b183-66e0da518d56" (UID: "91b95076-e971-49d4-b183-66e0da518d56"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:17:40.182677 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:40.182653    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91b95076-e971-49d4-b183-66e0da518d56-kube-api-access-nrrhh" pod "91b95076-e971-49d4-b183-66e0da518d56" (UID: "91b95076-e971-49d4-b183-66e0da518d56"). InnerVolumeSpecName "kube-api-access-nrrhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:17:40.278129 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:40.278097    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1720c00c-7f49c64bcf-dnv6v"]
Nov 03 18:17:40.281689 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:40.281644    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1720c00c-7f49c64bcf-dnv6v"]
Nov 03 18:17:40.281916 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:40.281889    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrrhh\" (UniqueName: \"kubernetes.io/projected/91b95076-e971-49d4-b183-66e0da518d56-kube-api-access-nrrhh\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:40.281966 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:40.281919    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ngk2\" (UniqueName: \"kubernetes.io/projected/e376e147-b415-47d0-8bb8-21aecf441da9-kube-api-access-9ngk2\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:40.281966 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:40.281929    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e376e147-b415-47d0-8bb8-21aecf441da9-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:40.281966 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:40.281939    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e376e147-b415-47d0-8bb8-21aecf441da9-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:40.281966 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:40.281948    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91b95076-e971-49d4-b183-66e0da518d56-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:40.281966 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:40.281957    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91b95076-e971-49d4-b183-66e0da518d56-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:40.828813 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:40.828777    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04d452e-wlv5h"
Nov 03 18:17:40.828813 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:40.828810    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04d452e-wlv5h" event={"ID":"e376e147-b415-47d0-8bb8-21aecf441da9","Type":"ContainerDied","Data":"47467a4278e916f71b319916e0eaf29699a676d6d4403e95bfae3b806ae84262"}
Nov 03 18:17:40.829284 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:40.828863    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47467a4278e916f71b319916e0eaf29699a676d6d4403e95bfae3b806ae84262"
Nov 03 18:17:40.832437 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:40.832408    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8941c20-8kr5x" event={"ID":"91b95076-e971-49d4-b183-66e0da518d56","Type":"ContainerDied","Data":"714c7b7811161f9a3a653d3d30642c2a9b09e686483e0cf03603bd06eed86085"}
Nov 03 18:17:40.832437 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:40.832436    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8941c20-8kr5x"
Nov 03 18:17:40.832601 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:40.832458    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="714c7b7811161f9a3a653d3d30642c2a9b09e686483e0cf03603bd06eed86085"
Nov 03 18:17:41.137024 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:41.136835    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c20ab79-c080-49c3-a84e-f8f3fc0713f2" path="/var/lib/kubelet/pods/2c20ab79-c080-49c3-a84e-f8f3fc0713f2/volumes"
Nov 03 18:17:41.137464 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:41.137436    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc4effb6-ba04-4eab-aeaf-a80ef45d2acb" path="/var/lib/kubelet/pods/fc4effb6-ba04-4eab-aeaf-a80ef45d2acb/volumes"
Nov 03 18:17:42.352274 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:42.352195    5315 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:17:42.352274 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:42.352250    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn" podUID="dafbc7c7-538e-4bf6-ab30-970870f21854" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:17:42.855394 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:42.855357    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1720c00c-24tp8"]
Nov 03 18:17:42.901418 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:42.901381    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1720c00c-24tp8"]
Nov 03 18:17:43.103881 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:43.103844    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abd83ea1-a9ec-42f3-a1f1-2943ff79d809" path="/var/lib/kubelet/pods/abd83ea1-a9ec-42f3-a1f1-2943ff79d809/volumes"
Nov 03 18:17:44.861467 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:44.861425    5315 generic.go:356] "Generic (PLEG): container finished" podID="dafbc7c7-538e-4bf6-ab30-970870f21854" containerID="131139a80d6fb7ad51d73854a2036d32c7d660b479bc2c84808e89e370b8592a" exitCode=0
Nov 03 18:17:44.861912 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:44.861503    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn" event={"ID":"dafbc7c7-538e-4bf6-ab30-970870f21854","Type":"ContainerDied","Data":"131139a80d6fb7ad51d73854a2036d32c7d660b479bc2c84808e89e370b8592a"}
Nov 03 18:17:44.864249 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:44.864222    5315 generic.go:356] "Generic (PLEG): container finished" podID="cb78e2af-5c83-4d6d-a562-274b12209c20" containerID="6da1959a991a4e6d878dc7df2369f8a6a53f12464d9442d98ed7737ca80b9eef" exitCode=0
Nov 03 18:17:44.864382 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:44.864304    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn" event={"ID":"cb78e2af-5c83-4d6d-a562-274b12209c20","Type":"ContainerDied","Data":"6da1959a991a4e6d878dc7df2369f8a6a53f12464d9442d98ed7737ca80b9eef"}
Nov 03 18:17:44.864458 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:44.864390    5315 scope.go:122] "RemoveContainer" containerID="ec2b17e223560519400e4f7691fa64d88d00bbd8d54624bfb48290322d244893"
Nov 03 18:17:44.988804 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:44.988760    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr" podUID="08159a02-fc2e-4138-824d-be5ba3e141d8" containerName="main" containerID="cri-o://23516f2c9599465b328d24128c5d8be7ac1a52e8eee5ef8bdecada9ebcd11397" gracePeriod=45
Nov 03 18:17:44.990657 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:44.990629    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn"
Nov 03 18:17:45.010185 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.010004    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l" podUID="114dcf54-e4eb-4294-a77b-604fe6d57d48" containerName="main" containerID="cri-o://48f26f116b804ff91ff27503181c6308ca41bba12d6db0880b978578d7d33193" gracePeriod=45
Nov 03 18:17:45.011147 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.011123    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn"
Nov 03 18:17:45.053313 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.053280    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dafbc7c7-538e-4bf6-ab30-970870f21854-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafbc7c7-538e-4bf6-ab30-970870f21854-credential-socket\") pod \"dafbc7c7-538e-4bf6-ab30-970870f21854\" (UID: \"dafbc7c7-538e-4bf6-ab30-970870f21854\") "
Nov 03 18:17:45.053425 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.053409    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dafbc7c7-538e-4bf6-ab30-970870f21854-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dafbc7c7-538e-4bf6-ab30-970870f21854-istio-envoy\") pod \"dafbc7c7-538e-4bf6-ab30-970870f21854\" (UID: \"dafbc7c7-538e-4bf6-ab30-970870f21854\") "
Nov 03 18:17:45.053513 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.053501    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dafbc7c7-538e-4bf6-ab30-970870f21854-kube-api-access-ngwk4\" (UniqueName: \"kubernetes.io/projected/dafbc7c7-538e-4bf6-ab30-970870f21854-kube-api-access-ngwk4\") pod \"dafbc7c7-538e-4bf6-ab30-970870f21854\" (UID: \"dafbc7c7-538e-4bf6-ab30-970870f21854\") "
Nov 03 18:17:45.053555 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.053530    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dafbc7c7-538e-4bf6-ab30-970870f21854-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafbc7c7-538e-4bf6-ab30-970870f21854-workload-socket\") pod \"dafbc7c7-538e-4bf6-ab30-970870f21854\" (UID: \"dafbc7c7-538e-4bf6-ab30-970870f21854\") "
Nov 03 18:17:45.053623 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.053558    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dafbc7c7-538e-4bf6-ab30-970870f21854-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dafbc7c7-538e-4bf6-ab30-970870f21854-istiod-ca-cert\") pod \"dafbc7c7-538e-4bf6-ab30-970870f21854\" (UID: \"dafbc7c7-538e-4bf6-ab30-970870f21854\") "
Nov 03 18:17:45.053623 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.053575    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dafbc7c7-538e-4bf6-ab30-970870f21854-credential-socket" pod "dafbc7c7-538e-4bf6-ab30-970870f21854" (UID: "dafbc7c7-538e-4bf6-ab30-970870f21854"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:45.053738 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.053588    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dafbc7c7-538e-4bf6-ab30-970870f21854-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dafbc7c7-538e-4bf6-ab30-970870f21854-istio-podinfo\") pod \"dafbc7c7-538e-4bf6-ab30-970870f21854\" (UID: \"dafbc7c7-538e-4bf6-ab30-970870f21854\") "
Nov 03 18:17:45.053738 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.053673    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dafbc7c7-538e-4bf6-ab30-970870f21854-istio-token\" (UniqueName: \"kubernetes.io/projected/dafbc7c7-538e-4bf6-ab30-970870f21854-istio-token\") pod \"dafbc7c7-538e-4bf6-ab30-970870f21854\" (UID: \"dafbc7c7-538e-4bf6-ab30-970870f21854\") "
Nov 03 18:17:45.053738 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.053705    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dafbc7c7-538e-4bf6-ab30-970870f21854-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dafbc7c7-538e-4bf6-ab30-970870f21854-workload-certs\") pod \"dafbc7c7-538e-4bf6-ab30-970870f21854\" (UID: \"dafbc7c7-538e-4bf6-ab30-970870f21854\") "
Nov 03 18:17:45.053891 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.053737    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dafbc7c7-538e-4bf6-ab30-970870f21854-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dafbc7c7-538e-4bf6-ab30-970870f21854-istio-data\") pod \"dafbc7c7-538e-4bf6-ab30-970870f21854\" (UID: \"dafbc7c7-538e-4bf6-ab30-970870f21854\") "
Nov 03 18:17:45.053945 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.053893    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dafbc7c7-538e-4bf6-ab30-970870f21854-workload-socket" pod "dafbc7c7-538e-4bf6-ab30-970870f21854" (UID: "dafbc7c7-538e-4bf6-ab30-970870f21854"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:45.054119 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.054097    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dafbc7c7-538e-4bf6-ab30-970870f21854-workload-certs" pod "dafbc7c7-538e-4bf6-ab30-970870f21854" (UID: "dafbc7c7-538e-4bf6-ab30-970870f21854"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:45.054338 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.054312    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dafbc7c7-538e-4bf6-ab30-970870f21854-istio-data" pod "dafbc7c7-538e-4bf6-ab30-970870f21854" (UID: "dafbc7c7-538e-4bf6-ab30-970870f21854"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:45.054418 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.054318    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafbc7c7-538e-4bf6-ab30-970870f21854-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:45.054418 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.054346    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dafbc7c7-538e-4bf6-ab30-970870f21854-istiod-ca-cert" pod "dafbc7c7-538e-4bf6-ab30-970870f21854" (UID: "dafbc7c7-538e-4bf6-ab30-970870f21854"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:17:45.054418 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.054364    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafbc7c7-538e-4bf6-ab30-970870f21854-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:45.054418 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.054381    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dafbc7c7-538e-4bf6-ab30-970870f21854-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:45.055927 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.055901    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dafbc7c7-538e-4bf6-ab30-970870f21854-istio-envoy" pod "dafbc7c7-538e-4bf6-ab30-970870f21854" (UID: "dafbc7c7-538e-4bf6-ab30-970870f21854"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:45.056039 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.055940    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dafbc7c7-538e-4bf6-ab30-970870f21854-kube-api-access-ngwk4" pod "dafbc7c7-538e-4bf6-ab30-970870f21854" (UID: "dafbc7c7-538e-4bf6-ab30-970870f21854"). InnerVolumeSpecName "kube-api-access-ngwk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:17:45.056237 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.056213    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dafbc7c7-538e-4bf6-ab30-970870f21854-istio-podinfo" pod "dafbc7c7-538e-4bf6-ab30-970870f21854" (UID: "dafbc7c7-538e-4bf6-ab30-970870f21854"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:17:45.056368 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.056341    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dafbc7c7-538e-4bf6-ab30-970870f21854-istio-token" pod "dafbc7c7-538e-4bf6-ab30-970870f21854" (UID: "dafbc7c7-538e-4bf6-ab30-970870f21854"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:17:45.157629 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.154867    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb78e2af-5c83-4d6d-a562-274b12209c20-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb78e2af-5c83-4d6d-a562-274b12209c20-workload-certs\") pod \"cb78e2af-5c83-4d6d-a562-274b12209c20\" (UID: \"cb78e2af-5c83-4d6d-a562-274b12209c20\") "
Nov 03 18:17:45.157629 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.154947    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb78e2af-5c83-4d6d-a562-274b12209c20-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb78e2af-5c83-4d6d-a562-274b12209c20-workload-socket\") pod \"cb78e2af-5c83-4d6d-a562-274b12209c20\" (UID: \"cb78e2af-5c83-4d6d-a562-274b12209c20\") "
Nov 03 18:17:45.157629 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.154987    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb78e2af-5c83-4d6d-a562-274b12209c20-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb78e2af-5c83-4d6d-a562-274b12209c20-istio-data\") pod \"cb78e2af-5c83-4d6d-a562-274b12209c20\" (UID: \"cb78e2af-5c83-4d6d-a562-274b12209c20\") "
Nov 03 18:17:45.157629 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.155051    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb78e2af-5c83-4d6d-a562-274b12209c20-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb78e2af-5c83-4d6d-a562-274b12209c20-istio-podinfo\") pod \"cb78e2af-5c83-4d6d-a562-274b12209c20\" (UID: \"cb78e2af-5c83-4d6d-a562-274b12209c20\") "
Nov 03 18:17:45.157629 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.155085    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb78e2af-5c83-4d6d-a562-274b12209c20-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb78e2af-5c83-4d6d-a562-274b12209c20-istio-envoy\") pod \"cb78e2af-5c83-4d6d-a562-274b12209c20\" (UID: \"cb78e2af-5c83-4d6d-a562-274b12209c20\") "
Nov 03 18:17:45.157629 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.155124    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb78e2af-5c83-4d6d-a562-274b12209c20-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb78e2af-5c83-4d6d-a562-274b12209c20-istiod-ca-cert\") pod \"cb78e2af-5c83-4d6d-a562-274b12209c20\" (UID: \"cb78e2af-5c83-4d6d-a562-274b12209c20\") "
Nov 03 18:17:45.157629 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.155193    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb78e2af-5c83-4d6d-a562-274b12209c20-kube-api-access-tv4nn\" (UniqueName: \"kubernetes.io/projected/cb78e2af-5c83-4d6d-a562-274b12209c20-kube-api-access-tv4nn\") pod \"cb78e2af-5c83-4d6d-a562-274b12209c20\" (UID: \"cb78e2af-5c83-4d6d-a562-274b12209c20\") "
Nov 03 18:17:45.157629 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.155223    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb78e2af-5c83-4d6d-a562-274b12209c20-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb78e2af-5c83-4d6d-a562-274b12209c20-credential-socket\") pod \"cb78e2af-5c83-4d6d-a562-274b12209c20\" (UID: \"cb78e2af-5c83-4d6d-a562-274b12209c20\") "
Nov 03 18:17:45.157629 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.155266    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb78e2af-5c83-4d6d-a562-274b12209c20-istio-token\" (UniqueName: \"kubernetes.io/projected/cb78e2af-5c83-4d6d-a562-274b12209c20-istio-token\") pod \"cb78e2af-5c83-4d6d-a562-274b12209c20\" (UID: \"cb78e2af-5c83-4d6d-a562-274b12209c20\") "
Nov 03 18:17:45.157629 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.155773    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dafbc7c7-538e-4bf6-ab30-970870f21854-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:45.157629 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.155790    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dafbc7c7-538e-4bf6-ab30-970870f21854-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:45.157629 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.155801    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dafbc7c7-538e-4bf6-ab30-970870f21854-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:45.157629 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.155810    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngwk4\" (UniqueName: \"kubernetes.io/projected/dafbc7c7-538e-4bf6-ab30-970870f21854-kube-api-access-ngwk4\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:45.157629 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.155819    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dafbc7c7-538e-4bf6-ab30-970870f21854-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:45.157629 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.155828    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dafbc7c7-538e-4bf6-ab30-970870f21854-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:45.157629 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.156281    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb78e2af-5c83-4d6d-a562-274b12209c20-workload-certs" pod "cb78e2af-5c83-4d6d-a562-274b12209c20" (UID: "cb78e2af-5c83-4d6d-a562-274b12209c20"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:45.157629 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.156434    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb78e2af-5c83-4d6d-a562-274b12209c20-workload-socket" pod "cb78e2af-5c83-4d6d-a562-274b12209c20" (UID: "cb78e2af-5c83-4d6d-a562-274b12209c20"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:45.157629 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.156550    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb78e2af-5c83-4d6d-a562-274b12209c20-istio-data" pod "cb78e2af-5c83-4d6d-a562-274b12209c20" (UID: "cb78e2af-5c83-4d6d-a562-274b12209c20"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:45.159323 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.158956    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb78e2af-5c83-4d6d-a562-274b12209c20-istio-podinfo" pod "cb78e2af-5c83-4d6d-a562-274b12209c20" (UID: "cb78e2af-5c83-4d6d-a562-274b12209c20"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:17:45.159323 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.159118    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb78e2af-5c83-4d6d-a562-274b12209c20-credential-socket" pod "cb78e2af-5c83-4d6d-a562-274b12209c20" (UID: "cb78e2af-5c83-4d6d-a562-274b12209c20"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:45.163177 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.161758    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb78e2af-5c83-4d6d-a562-274b12209c20-istiod-ca-cert" pod "cb78e2af-5c83-4d6d-a562-274b12209c20" (UID: "cb78e2af-5c83-4d6d-a562-274b12209c20"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:17:45.163771 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.163466    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb78e2af-5c83-4d6d-a562-274b12209c20-istio-envoy" pod "cb78e2af-5c83-4d6d-a562-274b12209c20" (UID: "cb78e2af-5c83-4d6d-a562-274b12209c20"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:45.163873 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.163796    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb78e2af-5c83-4d6d-a562-274b12209c20-istio-token" pod "cb78e2af-5c83-4d6d-a562-274b12209c20" (UID: "cb78e2af-5c83-4d6d-a562-274b12209c20"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:17:45.168653 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.166298    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb78e2af-5c83-4d6d-a562-274b12209c20-kube-api-access-tv4nn" pod "cb78e2af-5c83-4d6d-a562-274b12209c20" (UID: "cb78e2af-5c83-4d6d-a562-274b12209c20"). InnerVolumeSpecName "kube-api-access-tv4nn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:17:45.257651 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.257515    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb78e2af-5c83-4d6d-a562-274b12209c20-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:45.257651 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.257560    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb78e2af-5c83-4d6d-a562-274b12209c20-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:45.257651 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.257576    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb78e2af-5c83-4d6d-a562-274b12209c20-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:45.257651 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.257591    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb78e2af-5c83-4d6d-a562-274b12209c20-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:45.257651 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.257641    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb78e2af-5c83-4d6d-a562-274b12209c20-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:45.258082 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.257660    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv4nn\" (UniqueName: \"kubernetes.io/projected/cb78e2af-5c83-4d6d-a562-274b12209c20-kube-api-access-tv4nn\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:45.258082 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.257676    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb78e2af-5c83-4d6d-a562-274b12209c20-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:45.258082 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.257691    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb78e2af-5c83-4d6d-a562-274b12209c20-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:45.258082 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.257703    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb78e2af-5c83-4d6d-a562-274b12209c20-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:45.275867 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.275838    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr"
Nov 03 18:17:45.283542 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.283514    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l"
Nov 03 18:17:45.358731 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.358694    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08159a02-fc2e-4138-824d-be5ba3e141d8-kube-api-access-vgp2l\" (UniqueName: \"kubernetes.io/projected/08159a02-fc2e-4138-824d-be5ba3e141d8-kube-api-access-vgp2l\") pod \"08159a02-fc2e-4138-824d-be5ba3e141d8\" (UID: \"08159a02-fc2e-4138-824d-be5ba3e141d8\") "
Nov 03 18:17:45.358915 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.358747    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08159a02-fc2e-4138-824d-be5ba3e141d8-home\" (UniqueName: \"kubernetes.io/empty-dir/08159a02-fc2e-4138-824d-be5ba3e141d8-home\") pod \"08159a02-fc2e-4138-824d-be5ba3e141d8\" (UID: \"08159a02-fc2e-4138-824d-be5ba3e141d8\") "
Nov 03 18:17:45.358915 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.358772    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08159a02-fc2e-4138-824d-be5ba3e141d8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/08159a02-fc2e-4138-824d-be5ba3e141d8-dshm\") pod \"08159a02-fc2e-4138-824d-be5ba3e141d8\" (UID: \"08159a02-fc2e-4138-824d-be5ba3e141d8\") "
Nov 03 18:17:45.358915 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.358792    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/114dcf54-e4eb-4294-a77b-604fe6d57d48-kube-api-access-hnzpl\" (UniqueName: \"kubernetes.io/projected/114dcf54-e4eb-4294-a77b-604fe6d57d48-kube-api-access-hnzpl\") pod \"114dcf54-e4eb-4294-a77b-604fe6d57d48\" (UID: \"114dcf54-e4eb-4294-a77b-604fe6d57d48\") "
Nov 03 18:17:45.358915 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.358820    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08159a02-fc2e-4138-824d-be5ba3e141d8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08159a02-fc2e-4138-824d-be5ba3e141d8-model-cache\") pod \"08159a02-fc2e-4138-824d-be5ba3e141d8\" (UID: \"08159a02-fc2e-4138-824d-be5ba3e141d8\") "
Nov 03 18:17:45.358915 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.358862    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08159a02-fc2e-4138-824d-be5ba3e141d8-tls-certs\" (UniqueName: \"kubernetes.io/secret/08159a02-fc2e-4138-824d-be5ba3e141d8-tls-certs\") pod \"08159a02-fc2e-4138-824d-be5ba3e141d8\" (UID: \"08159a02-fc2e-4138-824d-be5ba3e141d8\") "
Nov 03 18:17:45.358915 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.358906    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/114dcf54-e4eb-4294-a77b-604fe6d57d48-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/114dcf54-e4eb-4294-a77b-604fe6d57d48-model-cache\") pod \"114dcf54-e4eb-4294-a77b-604fe6d57d48\" (UID: \"114dcf54-e4eb-4294-a77b-604fe6d57d48\") "
Nov 03 18:17:45.358915 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.358939    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/114dcf54-e4eb-4294-a77b-604fe6d57d48-home\" (UniqueName: \"kubernetes.io/empty-dir/114dcf54-e4eb-4294-a77b-604fe6d57d48-home\") pod \"114dcf54-e4eb-4294-a77b-604fe6d57d48\" (UID: \"114dcf54-e4eb-4294-a77b-604fe6d57d48\") "
Nov 03 18:17:45.359297 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.358966    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08159a02-fc2e-4138-824d-be5ba3e141d8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08159a02-fc2e-4138-824d-be5ba3e141d8-tmp-dir\") pod \"08159a02-fc2e-4138-824d-be5ba3e141d8\" (UID: \"08159a02-fc2e-4138-824d-be5ba3e141d8\") "
Nov 03 18:17:45.359297 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.359086    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/114dcf54-e4eb-4294-a77b-604fe6d57d48-tls-certs\" (UniqueName: \"kubernetes.io/secret/114dcf54-e4eb-4294-a77b-604fe6d57d48-tls-certs\") pod \"114dcf54-e4eb-4294-a77b-604fe6d57d48\" (UID: \"114dcf54-e4eb-4294-a77b-604fe6d57d48\") "
Nov 03 18:17:45.359297 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.359102    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08159a02-fc2e-4138-824d-be5ba3e141d8-home" pod "08159a02-fc2e-4138-824d-be5ba3e141d8" (UID: "08159a02-fc2e-4138-824d-be5ba3e141d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:45.359297 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.359117    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08159a02-fc2e-4138-824d-be5ba3e141d8-model-cache" pod "08159a02-fc2e-4138-824d-be5ba3e141d8" (UID: "08159a02-fc2e-4138-824d-be5ba3e141d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:45.359297 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.359127    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/114dcf54-e4eb-4294-a77b-604fe6d57d48-dshm\" (UniqueName: \"kubernetes.io/empty-dir/114dcf54-e4eb-4294-a77b-604fe6d57d48-dshm\") pod \"114dcf54-e4eb-4294-a77b-604fe6d57d48\" (UID: \"114dcf54-e4eb-4294-a77b-604fe6d57d48\") "
Nov 03 18:17:45.359297 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.359193    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/114dcf54-e4eb-4294-a77b-604fe6d57d48-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/114dcf54-e4eb-4294-a77b-604fe6d57d48-tmp-dir\") pod \"114dcf54-e4eb-4294-a77b-604fe6d57d48\" (UID: \"114dcf54-e4eb-4294-a77b-604fe6d57d48\") "
Nov 03 18:17:45.359696 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.359300    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/114dcf54-e4eb-4294-a77b-604fe6d57d48-model-cache" pod "114dcf54-e4eb-4294-a77b-604fe6d57d48" (UID: "114dcf54-e4eb-4294-a77b-604fe6d57d48"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:45.359696 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.359319    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/114dcf54-e4eb-4294-a77b-604fe6d57d48-home" pod "114dcf54-e4eb-4294-a77b-604fe6d57d48" (UID: "114dcf54-e4eb-4294-a77b-604fe6d57d48"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:45.359696 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.359324    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08159a02-fc2e-4138-824d-be5ba3e141d8-tmp-dir" pod "08159a02-fc2e-4138-824d-be5ba3e141d8" (UID: "08159a02-fc2e-4138-824d-be5ba3e141d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:45.359696 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.359647    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/114dcf54-e4eb-4294-a77b-604fe6d57d48-tmp-dir" pod "114dcf54-e4eb-4294-a77b-604fe6d57d48" (UID: "114dcf54-e4eb-4294-a77b-604fe6d57d48"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:45.359696 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.359665    5315 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08159a02-fc2e-4138-824d-be5ba3e141d8-home\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:45.359696 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.359689    5315 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08159a02-fc2e-4138-824d-be5ba3e141d8-model-cache\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:45.360031 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.359706    5315 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/114dcf54-e4eb-4294-a77b-604fe6d57d48-model-cache\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:45.360031 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.359721    5315 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/114dcf54-e4eb-4294-a77b-604fe6d57d48-home\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:45.360031 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.359735    5315 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08159a02-fc2e-4138-824d-be5ba3e141d8-tmp-dir\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:45.361641 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.361442    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08159a02-fc2e-4138-824d-be5ba3e141d8-kube-api-access-vgp2l" pod "08159a02-fc2e-4138-824d-be5ba3e141d8" (UID: "08159a02-fc2e-4138-824d-be5ba3e141d8"). InnerVolumeSpecName "kube-api-access-vgp2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:17:45.361783 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.361697    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08159a02-fc2e-4138-824d-be5ba3e141d8-dshm" pod "08159a02-fc2e-4138-824d-be5ba3e141d8" (UID: "08159a02-fc2e-4138-824d-be5ba3e141d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:45.361903 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.361867    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/114dcf54-e4eb-4294-a77b-604fe6d57d48-dshm" pod "114dcf54-e4eb-4294-a77b-604fe6d57d48" (UID: "114dcf54-e4eb-4294-a77b-604fe6d57d48"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:45.361985 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.361899    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/114dcf54-e4eb-4294-a77b-604fe6d57d48-tls-certs" pod "114dcf54-e4eb-4294-a77b-604fe6d57d48" (UID: "114dcf54-e4eb-4294-a77b-604fe6d57d48"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:17:45.362046 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.361978    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/114dcf54-e4eb-4294-a77b-604fe6d57d48-kube-api-access-hnzpl" pod "114dcf54-e4eb-4294-a77b-604fe6d57d48" (UID: "114dcf54-e4eb-4294-a77b-604fe6d57d48"). InnerVolumeSpecName "kube-api-access-hnzpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:17:45.362194 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.362178    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08159a02-fc2e-4138-824d-be5ba3e141d8-tls-certs" pod "08159a02-fc2e-4138-824d-be5ba3e141d8" (UID: "08159a02-fc2e-4138-824d-be5ba3e141d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:17:45.460852 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.460818    5315 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/114dcf54-e4eb-4294-a77b-604fe6d57d48-tls-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:45.460852 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.460846    5315 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/114dcf54-e4eb-4294-a77b-604fe6d57d48-dshm\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:45.460852 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.460856    5315 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/114dcf54-e4eb-4294-a77b-604fe6d57d48-tmp-dir\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:45.461091 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.460865    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgp2l\" (UniqueName: \"kubernetes.io/projected/08159a02-fc2e-4138-824d-be5ba3e141d8-kube-api-access-vgp2l\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:45.461091 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.460875    5315 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08159a02-fc2e-4138-824d-be5ba3e141d8-dshm\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:45.461091 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.460886    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnzpl\" (UniqueName: \"kubernetes.io/projected/114dcf54-e4eb-4294-a77b-604fe6d57d48-kube-api-access-hnzpl\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:45.461091 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.460895    5315 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08159a02-fc2e-4138-824d-be5ba3e141d8-tls-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:45.872583 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.872549    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn"
Nov 03 18:17:45.873072 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.872549    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn" event={"ID":"dafbc7c7-538e-4bf6-ab30-970870f21854","Type":"ContainerDied","Data":"2ec0a319d634e0b08756b134f2e00e9216500a1de1f43e7c4b084db39f7ab4d6"}
Nov 03 18:17:45.873072 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.872688    5315 scope.go:122] "RemoveContainer" containerID="131139a80d6fb7ad51d73854a2036d32c7d660b479bc2c84808e89e370b8592a"
Nov 03 18:17:45.874098 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.874057    5315 generic.go:356] "Generic (PLEG): container finished" podID="114dcf54-e4eb-4294-a77b-604fe6d57d48" containerID="48f26f116b804ff91ff27503181c6308ca41bba12d6db0880b978578d7d33193" exitCode=0
Nov 03 18:17:45.874196 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.874112    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l"
Nov 03 18:17:45.874196 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.874145    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l" event={"ID":"114dcf54-e4eb-4294-a77b-604fe6d57d48","Type":"ContainerDied","Data":"48f26f116b804ff91ff27503181c6308ca41bba12d6db0880b978578d7d33193"}
Nov 03 18:17:45.874196 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.874187    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l" event={"ID":"114dcf54-e4eb-4294-a77b-604fe6d57d48","Type":"ContainerDied","Data":"10772cdde472eb8b664c4876fdf69760ae1ef41c8a6ecaff2b04a4068c0b264e"}
Nov 03 18:17:45.875941 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.875913    5315 generic.go:356] "Generic (PLEG): container finished" podID="08159a02-fc2e-4138-824d-be5ba3e141d8" containerID="23516f2c9599465b328d24128c5d8be7ac1a52e8eee5ef8bdecada9ebcd11397" exitCode=0
Nov 03 18:17:45.876046 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.875949    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr"
Nov 03 18:17:45.876046 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.876002    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr" event={"ID":"08159a02-fc2e-4138-824d-be5ba3e141d8","Type":"ContainerDied","Data":"23516f2c9599465b328d24128c5d8be7ac1a52e8eee5ef8bdecada9ebcd11397"}
Nov 03 18:17:45.876046 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.876043    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr" event={"ID":"08159a02-fc2e-4138-824d-be5ba3e141d8","Type":"ContainerDied","Data":"8e264a1b677f12a4408792bdae6abfa88226dcabfc21c9e1d63cd4441860665b"}
Nov 03 18:17:45.877686 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.877665    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn"
Nov 03 18:17:45.877783 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.877705    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn" event={"ID":"cb78e2af-5c83-4d6d-a562-274b12209c20","Type":"ContainerDied","Data":"ff72f586bdc097b0d351cb419dfab78ceff515a5b33566e1951f64dedb078c1d"}
Nov 03 18:17:45.885929 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.885821    5315 scope.go:122] "RemoveContainer" containerID="48f26f116b804ff91ff27503181c6308ca41bba12d6db0880b978578d7d33193"
Nov 03 18:17:45.895701 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.895675    5315 scope.go:122] "RemoveContainer" containerID="48f26f116b804ff91ff27503181c6308ca41bba12d6db0880b978578d7d33193"
Nov 03 18:17:45.896007 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:17:45.895983    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48f26f116b804ff91ff27503181c6308ca41bba12d6db0880b978578d7d33193\": container with ID starting with 48f26f116b804ff91ff27503181c6308ca41bba12d6db0880b978578d7d33193 not found: ID does not exist" containerID="48f26f116b804ff91ff27503181c6308ca41bba12d6db0880b978578d7d33193"
Nov 03 18:17:45.896056 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.896017    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48f26f116b804ff91ff27503181c6308ca41bba12d6db0880b978578d7d33193"} err="failed to get container status \"48f26f116b804ff91ff27503181c6308ca41bba12d6db0880b978578d7d33193\": rpc error: code = NotFound desc = could not find container \"48f26f116b804ff91ff27503181c6308ca41bba12d6db0880b978578d7d33193\": container with ID starting with 48f26f116b804ff91ff27503181c6308ca41bba12d6db0880b978578d7d33193 not found: ID does not exist"
Nov 03 18:17:45.896056 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.896035    5315 scope.go:122] "RemoveContainer" containerID="23516f2c9599465b328d24128c5d8be7ac1a52e8eee5ef8bdecada9ebcd11397"
Nov 03 18:17:45.905225 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.905204    5315 scope.go:122] "RemoveContainer" containerID="23516f2c9599465b328d24128c5d8be7ac1a52e8eee5ef8bdecada9ebcd11397"
Nov 03 18:17:45.905521 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:17:45.905496    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23516f2c9599465b328d24128c5d8be7ac1a52e8eee5ef8bdecada9ebcd11397\": container with ID starting with 23516f2c9599465b328d24128c5d8be7ac1a52e8eee5ef8bdecada9ebcd11397 not found: ID does not exist" containerID="23516f2c9599465b328d24128c5d8be7ac1a52e8eee5ef8bdecada9ebcd11397"
Nov 03 18:17:45.905648 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.905528    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23516f2c9599465b328d24128c5d8be7ac1a52e8eee5ef8bdecada9ebcd11397"} err="failed to get container status \"23516f2c9599465b328d24128c5d8be7ac1a52e8eee5ef8bdecada9ebcd11397\": rpc error: code = NotFound desc = could not find container \"23516f2c9599465b328d24128c5d8be7ac1a52e8eee5ef8bdecada9ebcd11397\": container with ID starting with 23516f2c9599465b328d24128c5d8be7ac1a52e8eee5ef8bdecada9ebcd11397 not found: ID does not exist"
Nov 03 18:17:45.905648 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:45.905547    5315 scope.go:122] "RemoveContainer" containerID="6da1959a991a4e6d878dc7df2369f8a6a53f12464d9442d98ed7737ca80b9eef"
Nov 03 18:17:46.025581 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:46.025484    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn"]
Nov 03 18:17:46.030775 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:46.030745    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bf2nn"]
Nov 03 18:17:46.132063 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:46.132031    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn"]
Nov 03 18:17:46.137260 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:46.137232    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-bttqn"]
Nov 03 18:17:46.238079 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:46.238042    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l"]
Nov 03 18:17:46.240116 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:46.240090    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-2b4f46/simulator-w5-2b4f46-kserve-67884bc556-bsf5l"]
Nov 03 18:17:46.340780 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:46.340742    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr"]
Nov 03 18:17:46.346838 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:46.346807    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-2b4f46/distinct-2-w5-2b4f46-kserve-5c8f7cb99f-hczsr"]
Nov 03 18:17:47.109035 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:47.108996    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08159a02-fc2e-4138-824d-be5ba3e141d8" path="/var/lib/kubelet/pods/08159a02-fc2e-4138-824d-be5ba3e141d8/volumes"
Nov 03 18:17:47.109639 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:47.109588    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="114dcf54-e4eb-4294-a77b-604fe6d57d48" path="/var/lib/kubelet/pods/114dcf54-e4eb-4294-a77b-604fe6d57d48/volumes"
Nov 03 18:17:47.110106 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:47.110086    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb78e2af-5c83-4d6d-a562-274b12209c20" path="/var/lib/kubelet/pods/cb78e2af-5c83-4d6d-a562-274b12209c20/volumes"
Nov 03 18:17:47.110906 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:47.110884    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dafbc7c7-538e-4bf6-ab30-970870f21854" path="/var/lib/kubelet/pods/dafbc7c7-538e-4bf6-ab30-970870f21854/volumes"
Nov 03 18:17:48.349151 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:48.349111    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-1720c00c/e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr"]
Nov 03 18:17:53.260760 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:53.260718    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2b4f46-jt8sd"]
Nov 03 18:17:53.306645 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:53.306582    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2b4f46-jt8sd"]
Nov 03 18:17:54.577444 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:17:54.577399    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:17:55.103500 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:55.103468    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7bca6e3-4297-4b19-9972-a6f444725f6e" path="/var/lib/kubelet/pods/d7bca6e3-4297-4b19-9972-a6f444725f6e/volumes"
Nov 03 18:17:59.381329 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:59.381259    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j"]
Nov 03 18:17:59.381863 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:59.381624    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j" podUID="205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc" containerName="istio-proxy" containerID="cri-o://22a056e2c7fd07e28d24035fd8cfb51f6d6907ec08b2f74009bcf3608a4c7a95" gracePeriod=30
Nov 03 18:17:59.396768 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:59.396725    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg"]
Nov 03 18:17:59.396988 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:59.396965    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg" podUID="cb937207-ced4-4180-a554-0b7a86edce58" containerName="istio-proxy" containerID="cri-o://71e81734417c837c91e7b529296f44d2f7708dd73f25b9578a33d845895f0ba9" gracePeriod=30
Nov 03 18:17:59.398733 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:59.398709    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6"]
Nov 03 18:17:59.398943 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:59.398920    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6" podUID="6511822b-0252-4d76-88c0-b4e27a0d2e6e" containerName="istio-proxy" containerID="cri-o://7620550b4f502e4495db5d545e3614b0bf45ffe303b473e8d496eed5c5fe6eae" gracePeriod=30
Nov 03 18:17:59.442808 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:59.442771    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89"]
Nov 03 18:17:59.443021 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:17:59.442999    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89" podUID="16411400-4aa7-4754-9dc2-82175924011a" containerName="istio-proxy" containerID="cri-o://5c9d793dda22c3f9f174984911ac0248444bfb8cf3ac61f4e371214658b2a8d8" gracePeriod=30
Nov 03 18:18:00.690202 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:00.690166    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp"]
Nov 03 18:18:00.690690 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:00.690410    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp" podUID="3cf05bbd-6605-4cd2-aa94-85cbb46223aa" containerName="istio-proxy" containerID="cri-o://e6ac8fe445fba2f76fe442103c503bce99e901c53b27136bc55574863ff943d0" gracePeriod=30
Nov 03 18:18:00.700868 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:00.700834    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx"]
Nov 03 18:18:00.701104 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:00.701076    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx" podUID="d3aa0994-51ec-4076-bc86-c529ad0bb993" containerName="istio-proxy" containerID="cri-o://f483f014e8dc90ff635ed6820c3298a9734aefa3720a84ca1339fe199273932d" gracePeriod=30
Nov 03 18:18:03.369443 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:03.369384    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-1720c00c/e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr" podUID="8b7d3471-e6a0-4514-bf01-afb303808e53" containerName="main" containerID="cri-o://b9ce47857130ff14719d41d7572ab8e0102fb6d7dcd26e99cc718bbb40a6f8e6" gracePeriod=45
Nov 03 18:18:03.414176 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:03.414132    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-1720c00c/e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr" podUID="8b7d3471-e6a0-4514-bf01-afb303808e53" containerName="main" probeResult="failure" output="Get \"https://10.133.0.167:8000/ready\": dial tcp 10.133.0.167:8000: connect: connection refused"
Nov 03 18:18:03.624278 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:03.624219    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1720c00c/e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr"
Nov 03 18:18:03.690390 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:03.690355    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b7d3471-e6a0-4514-bf01-afb303808e53-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b7d3471-e6a0-4514-bf01-afb303808e53-tmp-dir\") pod \"8b7d3471-e6a0-4514-bf01-afb303808e53\" (UID: \"8b7d3471-e6a0-4514-bf01-afb303808e53\") "
Nov 03 18:18:03.690555 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:03.690442    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b7d3471-e6a0-4514-bf01-afb303808e53-tls-certs\" (UniqueName: \"kubernetes.io/secret/8b7d3471-e6a0-4514-bf01-afb303808e53-tls-certs\") pod \"8b7d3471-e6a0-4514-bf01-afb303808e53\" (UID: \"8b7d3471-e6a0-4514-bf01-afb303808e53\") "
Nov 03 18:18:03.690555 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:03.690498    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b7d3471-e6a0-4514-bf01-afb303808e53-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b7d3471-e6a0-4514-bf01-afb303808e53-model-cache\") pod \"8b7d3471-e6a0-4514-bf01-afb303808e53\" (UID: \"8b7d3471-e6a0-4514-bf01-afb303808e53\") "
Nov 03 18:18:03.690694 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:03.690557    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b7d3471-e6a0-4514-bf01-afb303808e53-kube-api-access-k2xkn\" (UniqueName: \"kubernetes.io/projected/8b7d3471-e6a0-4514-bf01-afb303808e53-kube-api-access-k2xkn\") pod \"8b7d3471-e6a0-4514-bf01-afb303808e53\" (UID: \"8b7d3471-e6a0-4514-bf01-afb303808e53\") "
Nov 03 18:18:03.690694 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:03.690631    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b7d3471-e6a0-4514-bf01-afb303808e53-home\" (UniqueName: \"kubernetes.io/empty-dir/8b7d3471-e6a0-4514-bf01-afb303808e53-home\") pod \"8b7d3471-e6a0-4514-bf01-afb303808e53\" (UID: \"8b7d3471-e6a0-4514-bf01-afb303808e53\") "
Nov 03 18:18:03.690694 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:03.690671    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b7d3471-e6a0-4514-bf01-afb303808e53-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b7d3471-e6a0-4514-bf01-afb303808e53-dshm\") pod \"8b7d3471-e6a0-4514-bf01-afb303808e53\" (UID: \"8b7d3471-e6a0-4514-bf01-afb303808e53\") "
Nov 03 18:18:03.690870 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:03.690695    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b7d3471-e6a0-4514-bf01-afb303808e53-tmp-dir" pod "8b7d3471-e6a0-4514-bf01-afb303808e53" (UID: "8b7d3471-e6a0-4514-bf01-afb303808e53"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:03.690870 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:03.690807    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b7d3471-e6a0-4514-bf01-afb303808e53-model-cache" pod "8b7d3471-e6a0-4514-bf01-afb303808e53" (UID: "8b7d3471-e6a0-4514-bf01-afb303808e53"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:03.690988 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:03.690913    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b7d3471-e6a0-4514-bf01-afb303808e53-home" pod "8b7d3471-e6a0-4514-bf01-afb303808e53" (UID: "8b7d3471-e6a0-4514-bf01-afb303808e53"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:03.691081 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:03.691063    5315 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b7d3471-e6a0-4514-bf01-afb303808e53-model-cache\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:03.691137 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:03.691085    5315 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b7d3471-e6a0-4514-bf01-afb303808e53-home\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:03.691137 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:03.691102    5315 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b7d3471-e6a0-4514-bf01-afb303808e53-tmp-dir\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:03.692837 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:03.692808    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b7d3471-e6a0-4514-bf01-afb303808e53-dshm" pod "8b7d3471-e6a0-4514-bf01-afb303808e53" (UID: "8b7d3471-e6a0-4514-bf01-afb303808e53"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:03.692951 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:03.692811    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b7d3471-e6a0-4514-bf01-afb303808e53-tls-certs" pod "8b7d3471-e6a0-4514-bf01-afb303808e53" (UID: "8b7d3471-e6a0-4514-bf01-afb303808e53"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:18:03.692951 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:03.692850    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b7d3471-e6a0-4514-bf01-afb303808e53-kube-api-access-k2xkn" pod "8b7d3471-e6a0-4514-bf01-afb303808e53" (UID: "8b7d3471-e6a0-4514-bf01-afb303808e53"). InnerVolumeSpecName "kube-api-access-k2xkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:18:03.792685 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:03.792641    5315 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b7d3471-e6a0-4514-bf01-afb303808e53-tls-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:03.792685 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:03.792682    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2xkn\" (UniqueName: \"kubernetes.io/projected/8b7d3471-e6a0-4514-bf01-afb303808e53-kube-api-access-k2xkn\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:03.792685 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:03.792695    5315 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b7d3471-e6a0-4514-bf01-afb303808e53-dshm\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:03.872419 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:03.872374    5315 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:18:03.872592 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:03.872432    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx" podUID="d3aa0994-51ec-4076-bc86-c529ad0bb993" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:18:04.015877 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.015787    5315 generic.go:356] "Generic (PLEG): container finished" podID="8b7d3471-e6a0-4514-bf01-afb303808e53" containerID="b9ce47857130ff14719d41d7572ab8e0102fb6d7dcd26e99cc718bbb40a6f8e6" exitCode=0
Nov 03 18:18:04.015877 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.015844    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1720c00c/e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr"
Nov 03 18:18:04.016140 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.015871    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1720c00c/e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr" event={"ID":"8b7d3471-e6a0-4514-bf01-afb303808e53","Type":"ContainerDied","Data":"b9ce47857130ff14719d41d7572ab8e0102fb6d7dcd26e99cc718bbb40a6f8e6"}
Nov 03 18:18:04.016140 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.015921    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1720c00c/e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr" event={"ID":"8b7d3471-e6a0-4514-bf01-afb303808e53","Type":"ContainerDied","Data":"2880721b312f6820bdaed383f9034437cf086c0b5ec4ba555c947761212a5206"}
Nov 03 18:18:04.016140 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.015941    5315 scope.go:122] "RemoveContainer" containerID="b9ce47857130ff14719d41d7572ab8e0102fb6d7dcd26e99cc718bbb40a6f8e6"
Nov 03 18:18:04.026083 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.026061    5315 scope.go:122] "RemoveContainer" containerID="b9ce47857130ff14719d41d7572ab8e0102fb6d7dcd26e99cc718bbb40a6f8e6"
Nov 03 18:18:04.026323 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:18:04.026304    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9ce47857130ff14719d41d7572ab8e0102fb6d7dcd26e99cc718bbb40a6f8e6\": container with ID starting with b9ce47857130ff14719d41d7572ab8e0102fb6d7dcd26e99cc718bbb40a6f8e6 not found: ID does not exist" containerID="b9ce47857130ff14719d41d7572ab8e0102fb6d7dcd26e99cc718bbb40a6f8e6"
Nov 03 18:18:04.026379 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.026329    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9ce47857130ff14719d41d7572ab8e0102fb6d7dcd26e99cc718bbb40a6f8e6"} err="failed to get container status \"b9ce47857130ff14719d41d7572ab8e0102fb6d7dcd26e99cc718bbb40a6f8e6\": rpc error: code = NotFound desc = could not find container \"b9ce47857130ff14719d41d7572ab8e0102fb6d7dcd26e99cc718bbb40a6f8e6\": container with ID starting with b9ce47857130ff14719d41d7572ab8e0102fb6d7dcd26e99cc718bbb40a6f8e6 not found: ID does not exist"
Nov 03 18:18:04.167788 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.167750    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-1720c00c/e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr"]
Nov 03 18:18:04.171988 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.171956    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-1720c00c/e2e-lifecycle-model-1720c00c-kserve-656ffb9ff7-k47dr"]
Nov 03 18:18:04.262867 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.262823    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-g99pg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:18:04.263047 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.262897    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg" podUID="cb937207-ced4-4180-a554-0b7a86edce58" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:18:04.702150 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.702117    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6"
Nov 03 18:18:04.762306 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.762279    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg"
Nov 03 18:18:04.769693 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.769671    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j"
Nov 03 18:18:04.772961 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.772940    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89"
Nov 03 18:18:04.803596 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.803566    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-kube-api-access-5p2vg\" (UniqueName: \"kubernetes.io/projected/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-kube-api-access-5p2vg\") pod \"205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc\" (UID: \"205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc\") "
Nov 03 18:18:04.803749 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.803604    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6511822b-0252-4d76-88c0-b4e27a0d2e6e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6511822b-0252-4d76-88c0-b4e27a0d2e6e-istio-data\") pod \"6511822b-0252-4d76-88c0-b4e27a0d2e6e\" (UID: \"6511822b-0252-4d76-88c0-b4e27a0d2e6e\") "
Nov 03 18:18:04.803749 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.803661    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16411400-4aa7-4754-9dc2-82175924011a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16411400-4aa7-4754-9dc2-82175924011a-istio-data\") pod \"16411400-4aa7-4754-9dc2-82175924011a\" (UID: \"16411400-4aa7-4754-9dc2-82175924011a\") "
Nov 03 18:18:04.803749 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.803698    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6511822b-0252-4d76-88c0-b4e27a0d2e6e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6511822b-0252-4d76-88c0-b4e27a0d2e6e-workload-socket\") pod \"6511822b-0252-4d76-88c0-b4e27a0d2e6e\" (UID: \"6511822b-0252-4d76-88c0-b4e27a0d2e6e\") "
Nov 03 18:18:04.803749 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.803728    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6511822b-0252-4d76-88c0-b4e27a0d2e6e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6511822b-0252-4d76-88c0-b4e27a0d2e6e-istio-podinfo\") pod \"6511822b-0252-4d76-88c0-b4e27a0d2e6e\" (UID: \"6511822b-0252-4d76-88c0-b4e27a0d2e6e\") "
Nov 03 18:18:04.803971 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.803762    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb937207-ced4-4180-a554-0b7a86edce58-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb937207-ced4-4180-a554-0b7a86edce58-istiod-ca-cert\") pod \"cb937207-ced4-4180-a554-0b7a86edce58\" (UID: \"cb937207-ced4-4180-a554-0b7a86edce58\") "
Nov 03 18:18:04.803971 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.803796    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6511822b-0252-4d76-88c0-b4e27a0d2e6e-kube-api-access-6llt8\" (UniqueName: \"kubernetes.io/projected/6511822b-0252-4d76-88c0-b4e27a0d2e6e-kube-api-access-6llt8\") pod \"6511822b-0252-4d76-88c0-b4e27a0d2e6e\" (UID: \"6511822b-0252-4d76-88c0-b4e27a0d2e6e\") "
Nov 03 18:18:04.803971 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.803828    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb937207-ced4-4180-a554-0b7a86edce58-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb937207-ced4-4180-a554-0b7a86edce58-istio-podinfo\") pod \"cb937207-ced4-4180-a554-0b7a86edce58\" (UID: \"cb937207-ced4-4180-a554-0b7a86edce58\") "
Nov 03 18:18:04.803971 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.803855    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb937207-ced4-4180-a554-0b7a86edce58-kube-api-access-gwpld\" (UniqueName: \"kubernetes.io/projected/cb937207-ced4-4180-a554-0b7a86edce58-kube-api-access-gwpld\") pod \"cb937207-ced4-4180-a554-0b7a86edce58\" (UID: \"cb937207-ced4-4180-a554-0b7a86edce58\") "
Nov 03 18:18:04.803971 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.803887    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6511822b-0252-4d76-88c0-b4e27a0d2e6e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6511822b-0252-4d76-88c0-b4e27a0d2e6e-istiod-ca-cert\") pod \"6511822b-0252-4d76-88c0-b4e27a0d2e6e\" (UID: \"6511822b-0252-4d76-88c0-b4e27a0d2e6e\") "
Nov 03 18:18:04.803971 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.803942    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16411400-4aa7-4754-9dc2-82175924011a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16411400-4aa7-4754-9dc2-82175924011a-istiod-ca-cert\") pod \"16411400-4aa7-4754-9dc2-82175924011a\" (UID: \"16411400-4aa7-4754-9dc2-82175924011a\") "
Nov 03 18:18:04.804305 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.803972    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-credential-socket\") pod \"205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc\" (UID: \"205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc\") "
Nov 03 18:18:04.804305 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.804019    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb937207-ced4-4180-a554-0b7a86edce58-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb937207-ced4-4180-a554-0b7a86edce58-istio-envoy\") pod \"cb937207-ced4-4180-a554-0b7a86edce58\" (UID: \"cb937207-ced4-4180-a554-0b7a86edce58\") "
Nov 03 18:18:04.804305 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.804057    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb937207-ced4-4180-a554-0b7a86edce58-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb937207-ced4-4180-a554-0b7a86edce58-istio-data\") pod \"cb937207-ced4-4180-a554-0b7a86edce58\" (UID: \"cb937207-ced4-4180-a554-0b7a86edce58\") "
Nov 03 18:18:04.804305 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.804098    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb937207-ced4-4180-a554-0b7a86edce58-istio-token\" (UniqueName: \"kubernetes.io/projected/cb937207-ced4-4180-a554-0b7a86edce58-istio-token\") pod \"cb937207-ced4-4180-a554-0b7a86edce58\" (UID: \"cb937207-ced4-4180-a554-0b7a86edce58\") "
Nov 03 18:18:04.804305 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.804131    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-istio-data\") pod \"205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc\" (UID: \"205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc\") "
Nov 03 18:18:04.804305 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.804157    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb937207-ced4-4180-a554-0b7a86edce58-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb937207-ced4-4180-a554-0b7a86edce58-workload-certs\") pod \"cb937207-ced4-4180-a554-0b7a86edce58\" (UID: \"cb937207-ced4-4180-a554-0b7a86edce58\") "
Nov 03 18:18:04.804305 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.804190    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16411400-4aa7-4754-9dc2-82175924011a-istio-token\" (UniqueName: \"kubernetes.io/projected/16411400-4aa7-4754-9dc2-82175924011a-istio-token\") pod \"16411400-4aa7-4754-9dc2-82175924011a\" (UID: \"16411400-4aa7-4754-9dc2-82175924011a\") "
Nov 03 18:18:04.804305 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.804223    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16411400-4aa7-4754-9dc2-82175924011a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16411400-4aa7-4754-9dc2-82175924011a-istio-envoy\") pod \"16411400-4aa7-4754-9dc2-82175924011a\" (UID: \"16411400-4aa7-4754-9dc2-82175924011a\") "
Nov 03 18:18:04.804305 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.804277    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6511822b-0252-4d76-88c0-b4e27a0d2e6e-istio-token\" (UniqueName: \"kubernetes.io/projected/6511822b-0252-4d76-88c0-b4e27a0d2e6e-istio-token\") pod \"6511822b-0252-4d76-88c0-b4e27a0d2e6e\" (UID: \"6511822b-0252-4d76-88c0-b4e27a0d2e6e\") "
Nov 03 18:18:04.804806 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.804310    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16411400-4aa7-4754-9dc2-82175924011a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16411400-4aa7-4754-9dc2-82175924011a-workload-socket\") pod \"16411400-4aa7-4754-9dc2-82175924011a\" (UID: \"16411400-4aa7-4754-9dc2-82175924011a\") "
Nov 03 18:18:04.804806 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.804341    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-workload-socket\") pod \"205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc\" (UID: \"205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc\") "
Nov 03 18:18:04.804806 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.804378    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-istio-podinfo\") pod \"205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc\" (UID: \"205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc\") "
Nov 03 18:18:04.804806 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.804407    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16411400-4aa7-4754-9dc2-82175924011a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16411400-4aa7-4754-9dc2-82175924011a-workload-certs\") pod \"16411400-4aa7-4754-9dc2-82175924011a\" (UID: \"16411400-4aa7-4754-9dc2-82175924011a\") "
Nov 03 18:18:04.804806 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.804445    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16411400-4aa7-4754-9dc2-82175924011a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16411400-4aa7-4754-9dc2-82175924011a-credential-socket\") pod \"16411400-4aa7-4754-9dc2-82175924011a\" (UID: \"16411400-4aa7-4754-9dc2-82175924011a\") "
Nov 03 18:18:04.804806 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.804473    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-istio-envoy\") pod \"205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc\" (UID: \"205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc\") "
Nov 03 18:18:04.804806 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.804795    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16411400-4aa7-4754-9dc2-82175924011a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16411400-4aa7-4754-9dc2-82175924011a-istio-podinfo\") pod \"16411400-4aa7-4754-9dc2-82175924011a\" (UID: \"16411400-4aa7-4754-9dc2-82175924011a\") "
Nov 03 18:18:04.805944 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.804840    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-istiod-ca-cert\") pod \"205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc\" (UID: \"205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc\") "
Nov 03 18:18:04.805944 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.804906    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6511822b-0252-4d76-88c0-b4e27a0d2e6e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6511822b-0252-4d76-88c0-b4e27a0d2e6e-workload-certs\") pod \"6511822b-0252-4d76-88c0-b4e27a0d2e6e\" (UID: \"6511822b-0252-4d76-88c0-b4e27a0d2e6e\") "
Nov 03 18:18:04.805944 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.804935    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-workload-certs\") pod \"205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc\" (UID: \"205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc\") "
Nov 03 18:18:04.805944 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.804994    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6511822b-0252-4d76-88c0-b4e27a0d2e6e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6511822b-0252-4d76-88c0-b4e27a0d2e6e-credential-socket\") pod \"6511822b-0252-4d76-88c0-b4e27a0d2e6e\" (UID: \"6511822b-0252-4d76-88c0-b4e27a0d2e6e\") "
Nov 03 18:18:04.805944 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.805032    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6511822b-0252-4d76-88c0-b4e27a0d2e6e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6511822b-0252-4d76-88c0-b4e27a0d2e6e-istio-envoy\") pod \"6511822b-0252-4d76-88c0-b4e27a0d2e6e\" (UID: \"6511822b-0252-4d76-88c0-b4e27a0d2e6e\") "
Nov 03 18:18:04.805944 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.805071    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb937207-ced4-4180-a554-0b7a86edce58-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb937207-ced4-4180-a554-0b7a86edce58-credential-socket\") pod \"cb937207-ced4-4180-a554-0b7a86edce58\" (UID: \"cb937207-ced4-4180-a554-0b7a86edce58\") "
Nov 03 18:18:04.805944 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.805119    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16411400-4aa7-4754-9dc2-82175924011a-kube-api-access-8jrz6\" (UniqueName: \"kubernetes.io/projected/16411400-4aa7-4754-9dc2-82175924011a-kube-api-access-8jrz6\") pod \"16411400-4aa7-4754-9dc2-82175924011a\" (UID: \"16411400-4aa7-4754-9dc2-82175924011a\") "
Nov 03 18:18:04.805944 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.805151    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-istio-token\" (UniqueName: \"kubernetes.io/projected/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-istio-token\") pod \"205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc\" (UID: \"205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc\") "
Nov 03 18:18:04.805944 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.803970    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16411400-4aa7-4754-9dc2-82175924011a-istio-data" pod "16411400-4aa7-4754-9dc2-82175924011a" (UID: "16411400-4aa7-4754-9dc2-82175924011a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:04.805944 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.803981    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6511822b-0252-4d76-88c0-b4e27a0d2e6e-istio-data" pod "6511822b-0252-4d76-88c0-b4e27a0d2e6e" (UID: "6511822b-0252-4d76-88c0-b4e27a0d2e6e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:04.805944 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.803993    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6511822b-0252-4d76-88c0-b4e27a0d2e6e-workload-socket" pod "6511822b-0252-4d76-88c0-b4e27a0d2e6e" (UID: "6511822b-0252-4d76-88c0-b4e27a0d2e6e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:04.805944 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.804240    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb937207-ced4-4180-a554-0b7a86edce58-istiod-ca-cert" pod "cb937207-ced4-4180-a554-0b7a86edce58" (UID: "cb937207-ced4-4180-a554-0b7a86edce58"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:18:04.805944 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.804254    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6511822b-0252-4d76-88c0-b4e27a0d2e6e-istiod-ca-cert" pod "6511822b-0252-4d76-88c0-b4e27a0d2e6e" (UID: "6511822b-0252-4d76-88c0-b4e27a0d2e6e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:18:04.805944 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.804665    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-istio-data" pod "205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc" (UID: "205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:04.805944 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.804683    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb937207-ced4-4180-a554-0b7a86edce58-istio-data" pod "cb937207-ced4-4180-a554-0b7a86edce58" (UID: "cb937207-ced4-4180-a554-0b7a86edce58"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:04.805944 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.805018    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16411400-4aa7-4754-9dc2-82175924011a-workload-certs" pod "16411400-4aa7-4754-9dc2-82175924011a" (UID: "16411400-4aa7-4754-9dc2-82175924011a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:04.805944 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.804934    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16411400-4aa7-4754-9dc2-82175924011a-istiod-ca-cert" pod "16411400-4aa7-4754-9dc2-82175924011a" (UID: "16411400-4aa7-4754-9dc2-82175924011a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:18:04.805944 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.805211    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-workload-socket" pod "205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc" (UID: "205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:04.805944 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.805226    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-credential-socket" pod "205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc" (UID: "205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:04.805944 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.805513    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb937207-ced4-4180-a554-0b7a86edce58-workload-certs" pod "cb937207-ced4-4180-a554-0b7a86edce58" (UID: "cb937207-ced4-4180-a554-0b7a86edce58"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:04.805944 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.805647    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6511822b-0252-4d76-88c0-b4e27a0d2e6e-workload-certs" pod "6511822b-0252-4d76-88c0-b4e27a0d2e6e" (UID: "6511822b-0252-4d76-88c0-b4e27a0d2e6e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:04.805944 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.805850    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-workload-certs" pod "205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc" (UID: "205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:04.807213 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.806194    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16411400-4aa7-4754-9dc2-82175924011a-workload-socket" pod "16411400-4aa7-4754-9dc2-82175924011a" (UID: "16411400-4aa7-4754-9dc2-82175924011a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:04.807213 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.806892    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16411400-4aa7-4754-9dc2-82175924011a-credential-socket" pod "16411400-4aa7-4754-9dc2-82175924011a" (UID: "16411400-4aa7-4754-9dc2-82175924011a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:04.807337 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.807296    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6511822b-0252-4d76-88c0-b4e27a0d2e6e-credential-socket" pod "6511822b-0252-4d76-88c0-b4e27a0d2e6e" (UID: "6511822b-0252-4d76-88c0-b4e27a0d2e6e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:04.807461 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.807366    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb937207-ced4-4180-a554-0b7a86edce58-credential-socket" pod "cb937207-ced4-4180-a554-0b7a86edce58" (UID: "cb937207-ced4-4180-a554-0b7a86edce58"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:04.807654 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.807629    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb937207-ced4-4180-a554-0b7a86edce58-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb937207-ced4-4180-a554-0b7a86edce58-workload-socket\") pod \"cb937207-ced4-4180-a554-0b7a86edce58\" (UID: \"cb937207-ced4-4180-a554-0b7a86edce58\") "
Nov 03 18:18:04.807820 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.807794    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-istiod-ca-cert" pod "205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc" (UID: "205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:18:04.807986 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.807942    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb937207-ced4-4180-a554-0b7a86edce58-workload-socket" pod "cb937207-ced4-4180-a554-0b7a86edce58" (UID: "cb937207-ced4-4180-a554-0b7a86edce58"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:04.808578 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.808213    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb937207-ced4-4180-a554-0b7a86edce58-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:04.808578 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.808240    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb937207-ced4-4180-a554-0b7a86edce58-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:04.808578 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.808257    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6511822b-0252-4d76-88c0-b4e27a0d2e6e-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:04.808578 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.808272    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16411400-4aa7-4754-9dc2-82175924011a-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:04.808578 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.808287    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6511822b-0252-4d76-88c0-b4e27a0d2e6e-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:04.808578 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.808300    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb937207-ced4-4180-a554-0b7a86edce58-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:04.808578 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.808314    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6511822b-0252-4d76-88c0-b4e27a0d2e6e-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:04.808578 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.808330    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16411400-4aa7-4754-9dc2-82175924011a-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:04.808578 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.808330    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-kube-api-access-5p2vg" pod "205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc" (UID: "205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc"). InnerVolumeSpecName "kube-api-access-5p2vg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:18:04.808578 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.808345    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:04.808578 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.808360    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb937207-ced4-4180-a554-0b7a86edce58-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:04.808578 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.808375    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:04.808578 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.808388    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb937207-ced4-4180-a554-0b7a86edce58-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:04.808578 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.808405    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16411400-4aa7-4754-9dc2-82175924011a-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:04.808578 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.808420    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:04.808578 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.808435    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16411400-4aa7-4754-9dc2-82175924011a-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:04.808578 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.808450    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16411400-4aa7-4754-9dc2-82175924011a-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:04.808578 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.808465    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:04.808578 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.808481    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6511822b-0252-4d76-88c0-b4e27a0d2e6e-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:04.808578 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.808495    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:04.808578 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.808509    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6511822b-0252-4d76-88c0-b4e27a0d2e6e-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:04.809867 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.808962    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6511822b-0252-4d76-88c0-b4e27a0d2e6e-istio-token" pod "6511822b-0252-4d76-88c0-b4e27a0d2e6e" (UID: "6511822b-0252-4d76-88c0-b4e27a0d2e6e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:18:04.809867 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.809343    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6511822b-0252-4d76-88c0-b4e27a0d2e6e-istio-podinfo" pod "6511822b-0252-4d76-88c0-b4e27a0d2e6e" (UID: "6511822b-0252-4d76-88c0-b4e27a0d2e6e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:18:04.809867 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.809688    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-istio-token" pod "205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc" (UID: "205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:18:04.809867 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.809786    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb937207-ced4-4180-a554-0b7a86edce58-kube-api-access-gwpld" pod "cb937207-ced4-4180-a554-0b7a86edce58" (UID: "cb937207-ced4-4180-a554-0b7a86edce58"). InnerVolumeSpecName "kube-api-access-gwpld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:18:04.809867 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.809791    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb937207-ced4-4180-a554-0b7a86edce58-istio-podinfo" pod "cb937207-ced4-4180-a554-0b7a86edce58" (UID: "cb937207-ced4-4180-a554-0b7a86edce58"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:18:04.810187 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.809888    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6511822b-0252-4d76-88c0-b4e27a0d2e6e-kube-api-access-6llt8" pod "6511822b-0252-4d76-88c0-b4e27a0d2e6e" (UID: "6511822b-0252-4d76-88c0-b4e27a0d2e6e"). InnerVolumeSpecName "kube-api-access-6llt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:18:04.810187 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.810014    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-istio-envoy" pod "205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc" (UID: "205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:04.810187 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.810008    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb937207-ced4-4180-a554-0b7a86edce58-istio-token" pod "cb937207-ced4-4180-a554-0b7a86edce58" (UID: "cb937207-ced4-4180-a554-0b7a86edce58"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:18:04.810361 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.810337    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb937207-ced4-4180-a554-0b7a86edce58-istio-envoy" pod "cb937207-ced4-4180-a554-0b7a86edce58" (UID: "cb937207-ced4-4180-a554-0b7a86edce58"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:04.810417 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.810367    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-istio-podinfo" pod "205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc" (UID: "205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:18:04.810507 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.810468    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16411400-4aa7-4754-9dc2-82175924011a-istio-podinfo" pod "16411400-4aa7-4754-9dc2-82175924011a" (UID: "16411400-4aa7-4754-9dc2-82175924011a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:18:04.810507 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.810497    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16411400-4aa7-4754-9dc2-82175924011a-kube-api-access-8jrz6" pod "16411400-4aa7-4754-9dc2-82175924011a" (UID: "16411400-4aa7-4754-9dc2-82175924011a"). InnerVolumeSpecName "kube-api-access-8jrz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:18:04.810707 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.810569    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16411400-4aa7-4754-9dc2-82175924011a-istio-envoy" pod "16411400-4aa7-4754-9dc2-82175924011a" (UID: "16411400-4aa7-4754-9dc2-82175924011a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:04.810707 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.810695    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16411400-4aa7-4754-9dc2-82175924011a-istio-token" pod "16411400-4aa7-4754-9dc2-82175924011a" (UID: "16411400-4aa7-4754-9dc2-82175924011a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:18:04.810778 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.810770    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6511822b-0252-4d76-88c0-b4e27a0d2e6e-istio-envoy" pod "6511822b-0252-4d76-88c0-b4e27a0d2e6e" (UID: "6511822b-0252-4d76-88c0-b4e27a0d2e6e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:04.909244 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.909210    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p2vg\" (UniqueName: \"kubernetes.io/projected/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-kube-api-access-5p2vg\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:04.909244 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.909242    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6511822b-0252-4d76-88c0-b4e27a0d2e6e-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:04.909466 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.909253    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6llt8\" (UniqueName: \"kubernetes.io/projected/6511822b-0252-4d76-88c0-b4e27a0d2e6e-kube-api-access-6llt8\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:04.909466 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.909263    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb937207-ced4-4180-a554-0b7a86edce58-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:04.909466 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.909272    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwpld\" (UniqueName: \"kubernetes.io/projected/cb937207-ced4-4180-a554-0b7a86edce58-kube-api-access-gwpld\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:04.909466 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.909281    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb937207-ced4-4180-a554-0b7a86edce58-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:04.909466 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.909290    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb937207-ced4-4180-a554-0b7a86edce58-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:04.909466 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.909299    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16411400-4aa7-4754-9dc2-82175924011a-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:04.909466 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.909309    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16411400-4aa7-4754-9dc2-82175924011a-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:04.909466 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.909318    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6511822b-0252-4d76-88c0-b4e27a0d2e6e-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:04.909466 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.909328    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:04.909466 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.909338    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:04.909466 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.909347    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16411400-4aa7-4754-9dc2-82175924011a-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:04.909466 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.909356    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6511822b-0252-4d76-88c0-b4e27a0d2e6e-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:04.909466 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.909365    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jrz6\" (UniqueName: \"kubernetes.io/projected/16411400-4aa7-4754-9dc2-82175924011a-kube-api-access-8jrz6\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:04.909466 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.909373    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:04.978957 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.978873    5315 scope.go:122] "RemoveContainer" containerID="8fd5709be236f0591519b6d365dbfb7c00f5a186674198579b95d2ab1d8bb468"
Nov 03 18:18:04.988554 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.988532    5315 scope.go:122] "RemoveContainer" containerID="22a056e2c7fd07e28d24035fd8cfb51f6d6907ec08b2f74009bcf3608a4c7a95"
Nov 03 18:18:04.997524 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:04.997500    5315 scope.go:122] "RemoveContainer" containerID="71e81734417c837c91e7b529296f44d2f7708dd73f25b9578a33d845895f0ba9"
Nov 03 18:18:05.006216 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:05.006194    5315 scope.go:122] "RemoveContainer" containerID="7620550b4f502e4495db5d545e3614b0bf45ffe303b473e8d496eed5c5fe6eae"
Nov 03 18:18:05.014748 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:05.014722    5315 scope.go:122] "RemoveContainer" containerID="373421a227e145d568570e337731ca87dcd3a430773fb217f9ae78dc2982df23"
Nov 03 18:18:05.023479 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:05.023456    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6"
Nov 03 18:18:05.023599 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:05.023460    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6" event={"ID":"6511822b-0252-4d76-88c0-b4e27a0d2e6e","Type":"ContainerDied","Data":"7620550b4f502e4495db5d545e3614b0bf45ffe303b473e8d496eed5c5fe6eae"}
Nov 03 18:18:05.023599 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:05.023586    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6" event={"ID":"6511822b-0252-4d76-88c0-b4e27a0d2e6e","Type":"ContainerDied","Data":"6c1e310cc0667809c386471c2a422cbe98bd85e571da92b40c0731343857fc2b"}
Nov 03 18:18:05.023817 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:05.023662    5315 scope.go:122] "RemoveContainer" containerID="5c9d793dda22c3f9f174984911ac0248444bfb8cf3ac61f4e371214658b2a8d8"
Nov 03 18:18:05.025597 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:05.025579    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j"
Nov 03 18:18:05.025721 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:05.025643    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j" event={"ID":"205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc","Type":"ContainerDied","Data":"22a056e2c7fd07e28d24035fd8cfb51f6d6907ec08b2f74009bcf3608a4c7a95"}
Nov 03 18:18:05.025721 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:05.025687    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j" event={"ID":"205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc","Type":"ContainerDied","Data":"0010b07117fc436e0d949e411fc54e4733a3b607c9a28b207c77ee2ada2b58e6"}
Nov 03 18:18:05.027494 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:05.027468    5315 generic.go:356] "Generic (PLEG): container finished" podID="16411400-4aa7-4754-9dc2-82175924011a" containerID="5c9d793dda22c3f9f174984911ac0248444bfb8cf3ac61f4e371214658b2a8d8" exitCode=0
Nov 03 18:18:05.027599 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:05.027518    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89"
Nov 03 18:18:05.027599 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:05.027523    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89" event={"ID":"16411400-4aa7-4754-9dc2-82175924011a","Type":"ContainerDied","Data":"5c9d793dda22c3f9f174984911ac0248444bfb8cf3ac61f4e371214658b2a8d8"}
Nov 03 18:18:05.027599 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:05.027564    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89" event={"ID":"16411400-4aa7-4754-9dc2-82175924011a","Type":"ContainerDied","Data":"7b0cfa490a5bc46dbd6ad4fa8d7331d04d2be788758e382c74f55ad85440c91d"}
Nov 03 18:18:05.027808 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:05.027604    5315 scope.go:122] "RemoveContainer" containerID="5c9d793dda22c3f9f174984911ac0248444bfb8cf3ac61f4e371214658b2a8d8"
Nov 03 18:18:05.030375 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:05.030349    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg" event={"ID":"cb937207-ced4-4180-a554-0b7a86edce58","Type":"ContainerDied","Data":"71e81734417c837c91e7b529296f44d2f7708dd73f25b9578a33d845895f0ba9"}
Nov 03 18:18:05.030470 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:05.030390    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg"
Nov 03 18:18:05.030470 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:05.030400    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg" event={"ID":"cb937207-ced4-4180-a554-0b7a86edce58","Type":"ContainerDied","Data":"3799fcdbec47096a4d333434f2778d26e625d7578162bf7adeabb2086974eeb9"}
Nov 03 18:18:05.033231 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:05.033211    5315 scope.go:122] "RemoveContainer" containerID="5c9d793dda22c3f9f174984911ac0248444bfb8cf3ac61f4e371214658b2a8d8"
Nov 03 18:18:05.033508 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:18:05.033483    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c9d793dda22c3f9f174984911ac0248444bfb8cf3ac61f4e371214658b2a8d8\": container with ID starting with 5c9d793dda22c3f9f174984911ac0248444bfb8cf3ac61f4e371214658b2a8d8 not found: ID does not exist" containerID="5c9d793dda22c3f9f174984911ac0248444bfb8cf3ac61f4e371214658b2a8d8"
Nov 03 18:18:05.033573 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:05.033523    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c9d793dda22c3f9f174984911ac0248444bfb8cf3ac61f4e371214658b2a8d8"} err="failed to get container status \"5c9d793dda22c3f9f174984911ac0248444bfb8cf3ac61f4e371214658b2a8d8\": rpc error: code = NotFound desc = could not find container \"5c9d793dda22c3f9f174984911ac0248444bfb8cf3ac61f4e371214658b2a8d8\": container with ID starting with 5c9d793dda22c3f9f174984911ac0248444bfb8cf3ac61f4e371214658b2a8d8 not found: ID does not exist"
Nov 03 18:18:05.033743 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:18:05.033716    5315 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_istio-proxy_maas-default-gateway-openshift-default-6dcf757899-6gf89_openshift-ingress_16411400-4aa7-4754-9dc2-82175924011a_0 in pod sandbox 7b0cfa490a5bc46dbd6ad4fa8d7331d04d2be788758e382c74f55ad85440c91d from index: no such id: '5c9d793dda22c3f9f174984911ac0248444bfb8cf3ac61f4e371214658b2a8d8'" containerID="5c9d793dda22c3f9f174984911ac0248444bfb8cf3ac61f4e371214658b2a8d8"
Nov 03 18:18:05.033865 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:18:05.033752    5315 kuberuntime_gc.go:151] "Failed to remove container" err="rpc error: code = Unknown desc = failed to delete container k8s_istio-proxy_maas-default-gateway-openshift-default-6dcf757899-6gf89_openshift-ingress_16411400-4aa7-4754-9dc2-82175924011a_0 in pod sandbox 7b0cfa490a5bc46dbd6ad4fa8d7331d04d2be788758e382c74f55ad85440c91d from index: no such id: '5c9d793dda22c3f9f174984911ac0248444bfb8cf3ac61f4e371214658b2a8d8'" containerID="5c9d793dda22c3f9f174984911ac0248444bfb8cf3ac61f4e371214658b2a8d8"
Nov 03 18:18:05.102673 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:05.102651    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b7d3471-e6a0-4514-bf01-afb303808e53" path="/var/lib/kubelet/pods/8b7d3471-e6a0-4514-bf01-afb303808e53/volumes"
Nov 03 18:18:05.189191 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:05.189145    5315 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:18:05.189366 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:05.189208    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp" podUID="3cf05bbd-6605-4cd2-aa94-85cbb46223aa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:18:05.258317 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:05.258225    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6"]
Nov 03 18:18:05.262367 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:05.262337    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-q2lb6"]
Nov 03 18:18:05.364370 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:05.364330    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j"]
Nov 03 18:18:05.368056 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:05.368028    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6pq2j"]
Nov 03 18:18:05.470019 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:05.469984    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89"]
Nov 03 18:18:05.473536 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:05.473507    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6gf89"]
Nov 03 18:18:05.575746 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:05.575707    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg"]
Nov 03 18:18:05.579439 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:05.579413    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-g99pg"]
Nov 03 18:18:05.962780 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:05.962754    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp"
Nov 03 18:18:05.978525 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:05.978500    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx"
Nov 03 18:18:06.018715 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.018678    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3aa0994-51ec-4076-bc86-c529ad0bb993-kube-api-access-hkbsm\" (UniqueName: \"kubernetes.io/projected/d3aa0994-51ec-4076-bc86-c529ad0bb993-kube-api-access-hkbsm\") pod \"d3aa0994-51ec-4076-bc86-c529ad0bb993\" (UID: \"d3aa0994-51ec-4076-bc86-c529ad0bb993\") "
Nov 03 18:18:06.018715 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.018718    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-workload-certs\") pod \"3cf05bbd-6605-4cd2-aa94-85cbb46223aa\" (UID: \"3cf05bbd-6605-4cd2-aa94-85cbb46223aa\") "
Nov 03 18:18:06.018974 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.018742    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3aa0994-51ec-4076-bc86-c529ad0bb993-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3aa0994-51ec-4076-bc86-c529ad0bb993-istio-envoy\") pod \"d3aa0994-51ec-4076-bc86-c529ad0bb993\" (UID: \"d3aa0994-51ec-4076-bc86-c529ad0bb993\") "
Nov 03 18:18:06.018974 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.018772    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-istio-envoy\") pod \"3cf05bbd-6605-4cd2-aa94-85cbb46223aa\" (UID: \"3cf05bbd-6605-4cd2-aa94-85cbb46223aa\") "
Nov 03 18:18:06.018974 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.018799    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-istio-data\") pod \"3cf05bbd-6605-4cd2-aa94-85cbb46223aa\" (UID: \"3cf05bbd-6605-4cd2-aa94-85cbb46223aa\") "
Nov 03 18:18:06.018974 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.018821    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3aa0994-51ec-4076-bc86-c529ad0bb993-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3aa0994-51ec-4076-bc86-c529ad0bb993-workload-socket\") pod \"d3aa0994-51ec-4076-bc86-c529ad0bb993\" (UID: \"d3aa0994-51ec-4076-bc86-c529ad0bb993\") "
Nov 03 18:18:06.018974 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.018854    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3aa0994-51ec-4076-bc86-c529ad0bb993-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3aa0994-51ec-4076-bc86-c529ad0bb993-istiod-ca-cert\") pod \"d3aa0994-51ec-4076-bc86-c529ad0bb993\" (UID: \"d3aa0994-51ec-4076-bc86-c529ad0bb993\") "
Nov 03 18:18:06.018974 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.018884    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3aa0994-51ec-4076-bc86-c529ad0bb993-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3aa0994-51ec-4076-bc86-c529ad0bb993-istio-data\") pod \"d3aa0994-51ec-4076-bc86-c529ad0bb993\" (UID: \"d3aa0994-51ec-4076-bc86-c529ad0bb993\") "
Nov 03 18:18:06.018974 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.018916    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-istio-token\" (UniqueName: \"kubernetes.io/projected/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-istio-token\") pod \"3cf05bbd-6605-4cd2-aa94-85cbb46223aa\" (UID: \"3cf05bbd-6605-4cd2-aa94-85cbb46223aa\") "
Nov 03 18:18:06.018974 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.018954    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3aa0994-51ec-4076-bc86-c529ad0bb993-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3aa0994-51ec-4076-bc86-c529ad0bb993-workload-certs\") pod \"d3aa0994-51ec-4076-bc86-c529ad0bb993\" (UID: \"d3aa0994-51ec-4076-bc86-c529ad0bb993\") "
Nov 03 18:18:06.019393 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.018992    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-workload-socket\") pod \"3cf05bbd-6605-4cd2-aa94-85cbb46223aa\" (UID: \"3cf05bbd-6605-4cd2-aa94-85cbb46223aa\") "
Nov 03 18:18:06.019393 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.019050    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-istiod-ca-cert\") pod \"3cf05bbd-6605-4cd2-aa94-85cbb46223aa\" (UID: \"3cf05bbd-6605-4cd2-aa94-85cbb46223aa\") "
Nov 03 18:18:06.019393 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.019060    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-workload-certs" pod "3cf05bbd-6605-4cd2-aa94-85cbb46223aa" (UID: "3cf05bbd-6605-4cd2-aa94-85cbb46223aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:06.019393 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.019087    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-istio-podinfo\") pod \"3cf05bbd-6605-4cd2-aa94-85cbb46223aa\" (UID: \"3cf05bbd-6605-4cd2-aa94-85cbb46223aa\") "
Nov 03 18:18:06.019393 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.019125    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-credential-socket\") pod \"3cf05bbd-6605-4cd2-aa94-85cbb46223aa\" (UID: \"3cf05bbd-6605-4cd2-aa94-85cbb46223aa\") "
Nov 03 18:18:06.019393 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.019154    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-kube-api-access-jlh4w\" (UniqueName: \"kubernetes.io/projected/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-kube-api-access-jlh4w\") pod \"3cf05bbd-6605-4cd2-aa94-85cbb46223aa\" (UID: \"3cf05bbd-6605-4cd2-aa94-85cbb46223aa\") "
Nov 03 18:18:06.019393 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.019158    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3aa0994-51ec-4076-bc86-c529ad0bb993-workload-socket" pod "d3aa0994-51ec-4076-bc86-c529ad0bb993" (UID: "d3aa0994-51ec-4076-bc86-c529ad0bb993"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:06.019393 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.019156    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-istio-data" pod "3cf05bbd-6605-4cd2-aa94-85cbb46223aa" (UID: "3cf05bbd-6605-4cd2-aa94-85cbb46223aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:06.019393 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.019193    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d3aa0994-51ec-4076-bc86-c529ad0bb993-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3aa0994-51ec-4076-bc86-c529ad0bb993-istio-podinfo\") pod \"d3aa0994-51ec-4076-bc86-c529ad0bb993\" (UID: \"d3aa0994-51ec-4076-bc86-c529ad0bb993\") "
Nov 03 18:18:06.019393 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.019247    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3aa0994-51ec-4076-bc86-c529ad0bb993-istio-token\" (UniqueName: \"kubernetes.io/projected/d3aa0994-51ec-4076-bc86-c529ad0bb993-istio-token\") pod \"d3aa0994-51ec-4076-bc86-c529ad0bb993\" (UID: \"d3aa0994-51ec-4076-bc86-c529ad0bb993\") "
Nov 03 18:18:06.019393 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.019285    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3aa0994-51ec-4076-bc86-c529ad0bb993-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3aa0994-51ec-4076-bc86-c529ad0bb993-credential-socket\") pod \"d3aa0994-51ec-4076-bc86-c529ad0bb993\" (UID: \"d3aa0994-51ec-4076-bc86-c529ad0bb993\") "
Nov 03 18:18:06.019393 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.019330    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3aa0994-51ec-4076-bc86-c529ad0bb993-workload-certs" pod "d3aa0994-51ec-4076-bc86-c529ad0bb993" (UID: "d3aa0994-51ec-4076-bc86-c529ad0bb993"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:06.019393 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.019364    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-workload-socket" pod "3cf05bbd-6605-4cd2-aa94-85cbb46223aa" (UID: "3cf05bbd-6605-4cd2-aa94-85cbb46223aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:06.020113 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.019551    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-istiod-ca-cert" pod "3cf05bbd-6605-4cd2-aa94-85cbb46223aa" (UID: "3cf05bbd-6605-4cd2-aa94-85cbb46223aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:18:06.020113 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.019629    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:06.020113 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.019651    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:06.020113 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.019660    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3aa0994-51ec-4076-bc86-c529ad0bb993-istiod-ca-cert" pod "d3aa0994-51ec-4076-bc86-c529ad0bb993" (UID: "d3aa0994-51ec-4076-bc86-c529ad0bb993"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:18:06.020113 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.019666    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3aa0994-51ec-4076-bc86-c529ad0bb993-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:06.020113 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.019705    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3aa0994-51ec-4076-bc86-c529ad0bb993-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:06.020113 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.019722    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:06.020113 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.020009    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3aa0994-51ec-4076-bc86-c529ad0bb993-istio-data" pod "d3aa0994-51ec-4076-bc86-c529ad0bb993" (UID: "d3aa0994-51ec-4076-bc86-c529ad0bb993"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:06.020548 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.020230    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3aa0994-51ec-4076-bc86-c529ad0bb993-credential-socket" pod "d3aa0994-51ec-4076-bc86-c529ad0bb993" (UID: "d3aa0994-51ec-4076-bc86-c529ad0bb993"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:06.020548 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.020348    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-credential-socket" pod "3cf05bbd-6605-4cd2-aa94-85cbb46223aa" (UID: "3cf05bbd-6605-4cd2-aa94-85cbb46223aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:06.021559 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.021529    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3aa0994-51ec-4076-bc86-c529ad0bb993-istio-envoy" pod "d3aa0994-51ec-4076-bc86-c529ad0bb993" (UID: "d3aa0994-51ec-4076-bc86-c529ad0bb993"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:06.021854 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.021827    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3aa0994-51ec-4076-bc86-c529ad0bb993-kube-api-access-hkbsm" pod "d3aa0994-51ec-4076-bc86-c529ad0bb993" (UID: "d3aa0994-51ec-4076-bc86-c529ad0bb993"). InnerVolumeSpecName "kube-api-access-hkbsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:18:06.021974 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.021907    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-istio-envoy" pod "3cf05bbd-6605-4cd2-aa94-85cbb46223aa" (UID: "3cf05bbd-6605-4cd2-aa94-85cbb46223aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:06.022067 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.022045    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-istio-podinfo" pod "3cf05bbd-6605-4cd2-aa94-85cbb46223aa" (UID: "3cf05bbd-6605-4cd2-aa94-85cbb46223aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:18:06.022121 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.022052    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3aa0994-51ec-4076-bc86-c529ad0bb993-istio-podinfo" pod "d3aa0994-51ec-4076-bc86-c529ad0bb993" (UID: "d3aa0994-51ec-4076-bc86-c529ad0bb993"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:18:06.022258 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.022244    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-kube-api-access-jlh4w" pod "3cf05bbd-6605-4cd2-aa94-85cbb46223aa" (UID: "3cf05bbd-6605-4cd2-aa94-85cbb46223aa"). InnerVolumeSpecName "kube-api-access-jlh4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:18:06.022563 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.022551    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-istio-token" pod "3cf05bbd-6605-4cd2-aa94-85cbb46223aa" (UID: "3cf05bbd-6605-4cd2-aa94-85cbb46223aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:18:06.022800 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.022788    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3aa0994-51ec-4076-bc86-c529ad0bb993-istio-token" pod "d3aa0994-51ec-4076-bc86-c529ad0bb993" (UID: "d3aa0994-51ec-4076-bc86-c529ad0bb993"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:18:06.038023 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.037996    5315 generic.go:356] "Generic (PLEG): container finished" podID="d3aa0994-51ec-4076-bc86-c529ad0bb993" containerID="f483f014e8dc90ff635ed6820c3298a9734aefa3720a84ca1339fe199273932d" exitCode=0
Nov 03 18:18:06.038190 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.038047    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx"
Nov 03 18:18:06.038190 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.038074    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx" event={"ID":"d3aa0994-51ec-4076-bc86-c529ad0bb993","Type":"ContainerDied","Data":"f483f014e8dc90ff635ed6820c3298a9734aefa3720a84ca1339fe199273932d"}
Nov 03 18:18:06.038190 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.038122    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx" event={"ID":"d3aa0994-51ec-4076-bc86-c529ad0bb993","Type":"ContainerDied","Data":"84189e3f98f05c87b0d924fbb6e61df08aba83a275fe58c4962ba38ea9f8b61b"}
Nov 03 18:18:06.038190 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.038157    5315 scope.go:122] "RemoveContainer" containerID="f483f014e8dc90ff635ed6820c3298a9734aefa3720a84ca1339fe199273932d"
Nov 03 18:18:06.039588 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.039567    5315 generic.go:356] "Generic (PLEG): container finished" podID="3cf05bbd-6605-4cd2-aa94-85cbb46223aa" containerID="e6ac8fe445fba2f76fe442103c503bce99e901c53b27136bc55574863ff943d0" exitCode=0
Nov 03 18:18:06.039699 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.039672    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp"
Nov 03 18:18:06.039766 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.039729    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp" event={"ID":"3cf05bbd-6605-4cd2-aa94-85cbb46223aa","Type":"ContainerDied","Data":"e6ac8fe445fba2f76fe442103c503bce99e901c53b27136bc55574863ff943d0"}
Nov 03 18:18:06.039805 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.039772    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp" event={"ID":"3cf05bbd-6605-4cd2-aa94-85cbb46223aa","Type":"ContainerDied","Data":"bfdea99b678db1b092e993d51d7b7f6da7baf8b67514794c92f5fa9189e5aa8f"}
Nov 03 18:18:06.049628 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.049594    5315 scope.go:122] "RemoveContainer" containerID="f483f014e8dc90ff635ed6820c3298a9734aefa3720a84ca1339fe199273932d"
Nov 03 18:18:06.049969 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:18:06.049946    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f483f014e8dc90ff635ed6820c3298a9734aefa3720a84ca1339fe199273932d\": container with ID starting with f483f014e8dc90ff635ed6820c3298a9734aefa3720a84ca1339fe199273932d not found: ID does not exist" containerID="f483f014e8dc90ff635ed6820c3298a9734aefa3720a84ca1339fe199273932d"
Nov 03 18:18:06.050055 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.049987    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f483f014e8dc90ff635ed6820c3298a9734aefa3720a84ca1339fe199273932d"} err="failed to get container status \"f483f014e8dc90ff635ed6820c3298a9734aefa3720a84ca1339fe199273932d\": rpc error: code = NotFound desc = could not find container \"f483f014e8dc90ff635ed6820c3298a9734aefa3720a84ca1339fe199273932d\": container with ID starting with f483f014e8dc90ff635ed6820c3298a9734aefa3720a84ca1339fe199273932d not found: ID does not exist"
Nov 03 18:18:06.050055 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.050014    5315 scope.go:122] "RemoveContainer" containerID="e6ac8fe445fba2f76fe442103c503bce99e901c53b27136bc55574863ff943d0"
Nov 03 18:18:06.058653 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.058629    5315 scope.go:122] "RemoveContainer" containerID="e6ac8fe445fba2f76fe442103c503bce99e901c53b27136bc55574863ff943d0"
Nov 03 18:18:06.058927 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:18:06.058907    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6ac8fe445fba2f76fe442103c503bce99e901c53b27136bc55574863ff943d0\": container with ID starting with e6ac8fe445fba2f76fe442103c503bce99e901c53b27136bc55574863ff943d0 not found: ID does not exist" containerID="e6ac8fe445fba2f76fe442103c503bce99e901c53b27136bc55574863ff943d0"
Nov 03 18:18:06.058990 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.058937    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6ac8fe445fba2f76fe442103c503bce99e901c53b27136bc55574863ff943d0"} err="failed to get container status \"e6ac8fe445fba2f76fe442103c503bce99e901c53b27136bc55574863ff943d0\": rpc error: code = NotFound desc = could not find container \"e6ac8fe445fba2f76fe442103c503bce99e901c53b27136bc55574863ff943d0\": container with ID starting with e6ac8fe445fba2f76fe442103c503bce99e901c53b27136bc55574863ff943d0 not found: ID does not exist"
Nov 03 18:18:06.120568 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.120492    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3aa0994-51ec-4076-bc86-c529ad0bb993-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:06.120568 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.120523    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3aa0994-51ec-4076-bc86-c529ad0bb993-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:06.120568 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.120532    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:06.120568 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.120540    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:06.120568 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.120549    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:06.120568 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.120557    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:06.120568 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.120567    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlh4w\" (UniqueName: \"kubernetes.io/projected/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-kube-api-access-jlh4w\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:06.120926 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.120578    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3aa0994-51ec-4076-bc86-c529ad0bb993-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:06.120926 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.120586    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3aa0994-51ec-4076-bc86-c529ad0bb993-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:06.120926 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.120596    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3aa0994-51ec-4076-bc86-c529ad0bb993-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:06.120926 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.120604    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkbsm\" (UniqueName: \"kubernetes.io/projected/d3aa0994-51ec-4076-bc86-c529ad0bb993-kube-api-access-hkbsm\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:06.120926 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.120629    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3aa0994-51ec-4076-bc86-c529ad0bb993-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:06.120926 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.120637    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cf05bbd-6605-4cd2-aa94-85cbb46223aa-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:06.180650 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.180587    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp"]
Nov 03 18:18:06.185746 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.185719    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-flwrp"]
Nov 03 18:18:06.355622 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.355569    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx"]
Nov 03 18:18:06.360937 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:06.360909    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-t7zwx"]
Nov 03 18:18:07.103556 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:07.103521    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16411400-4aa7-4754-9dc2-82175924011a" path="/var/lib/kubelet/pods/16411400-4aa7-4754-9dc2-82175924011a/volumes"
Nov 03 18:18:07.103989 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:07.103974    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc" path="/var/lib/kubelet/pods/205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc/volumes"
Nov 03 18:18:07.104365 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:07.104353    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cf05bbd-6605-4cd2-aa94-85cbb46223aa" path="/var/lib/kubelet/pods/3cf05bbd-6605-4cd2-aa94-85cbb46223aa/volumes"
Nov 03 18:18:07.105050 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:07.105032    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6511822b-0252-4d76-88c0-b4e27a0d2e6e" path="/var/lib/kubelet/pods/6511822b-0252-4d76-88c0-b4e27a0d2e6e/volumes"
Nov 03 18:18:07.105498 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:07.105483    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb937207-ced4-4180-a554-0b7a86edce58" path="/var/lib/kubelet/pods/cb937207-ced4-4180-a554-0b7a86edce58/volumes"
Nov 03 18:18:07.105996 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:07.105973    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3aa0994-51ec-4076-bc86-c529ad0bb993" path="/var/lib/kubelet/pods/d3aa0994-51ec-4076-bc86-c529ad0bb993/volumes"
Nov 03 18:18:24.576428 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:18:24.576386    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:18:37.783202 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:37.783165    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx"]
Nov 03 18:18:37.783823 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:37.783393    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx" podUID="6ebdce76-7e0e-4916-afcd-2aaf7cc44fed" containerName="istio-proxy" containerID="cri-o://7e1c5b05b3783b10ae91334682b73f908c0d459e6b3f486b75b5f62a1c055778" gracePeriod=30
Nov 03 18:18:43.046752 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:43.046683    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx"
Nov 03 18:18:43.065053 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:43.063441    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-istio-envoy\") pod \"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed\" (UID: \"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed\") "
Nov 03 18:18:43.065711 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:43.065678    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-istio-data\") pod \"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed\" (UID: \"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed\") "
Nov 03 18:18:43.065937 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:43.065912    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-credential-socket\") pod \"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed\" (UID: \"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed\") "
Nov 03 18:18:43.066084 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:43.065958    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-kube-api-access-bmngj\" (UniqueName: \"kubernetes.io/projected/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-kube-api-access-bmngj\") pod \"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed\" (UID: \"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed\") "
Nov 03 18:18:43.066084 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:43.065970    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-istio-data" pod "6ebdce76-7e0e-4916-afcd-2aaf7cc44fed" (UID: "6ebdce76-7e0e-4916-afcd-2aaf7cc44fed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:43.066084 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:43.066002    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-istio-token\" (UniqueName: \"kubernetes.io/projected/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-istio-token\") pod \"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed\" (UID: \"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed\") "
Nov 03 18:18:43.066295 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:43.066193    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-istio-podinfo\") pod \"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed\" (UID: \"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed\") "
Nov 03 18:18:43.066351 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:43.066307    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-workload-certs\") pod \"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed\" (UID: \"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed\") "
Nov 03 18:18:43.066414 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:43.066400    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-workload-socket\") pod \"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed\" (UID: \"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed\") "
Nov 03 18:18:43.066484 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:43.066439    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-istiod-ca-cert\") pod \"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed\" (UID: \"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed\") "
Nov 03 18:18:43.067675 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:43.066805    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:43.068222 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:43.066305    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-credential-socket" pod "6ebdce76-7e0e-4916-afcd-2aaf7cc44fed" (UID: "6ebdce76-7e0e-4916-afcd-2aaf7cc44fed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:43.068222 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:43.068168    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-workload-certs" pod "6ebdce76-7e0e-4916-afcd-2aaf7cc44fed" (UID: "6ebdce76-7e0e-4916-afcd-2aaf7cc44fed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:43.068437 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:43.068417    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-workload-socket" pod "6ebdce76-7e0e-4916-afcd-2aaf7cc44fed" (UID: "6ebdce76-7e0e-4916-afcd-2aaf7cc44fed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:43.068596 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:43.068542    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-istiod-ca-cert" pod "6ebdce76-7e0e-4916-afcd-2aaf7cc44fed" (UID: "6ebdce76-7e0e-4916-afcd-2aaf7cc44fed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:18:43.069144 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:43.069118    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-istio-envoy" pod "6ebdce76-7e0e-4916-afcd-2aaf7cc44fed" (UID: "6ebdce76-7e0e-4916-afcd-2aaf7cc44fed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:43.070714 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:43.070679    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-istio-podinfo" pod "6ebdce76-7e0e-4916-afcd-2aaf7cc44fed" (UID: "6ebdce76-7e0e-4916-afcd-2aaf7cc44fed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:18:43.073381 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:43.073352    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-kube-api-access-bmngj" pod "6ebdce76-7e0e-4916-afcd-2aaf7cc44fed" (UID: "6ebdce76-7e0e-4916-afcd-2aaf7cc44fed"). InnerVolumeSpecName "kube-api-access-bmngj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:18:43.078431 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:43.078402    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-istio-token" pod "6ebdce76-7e0e-4916-afcd-2aaf7cc44fed" (UID: "6ebdce76-7e0e-4916-afcd-2aaf7cc44fed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:18:43.168467 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:43.168426    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:43.168467 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:43.168469    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmngj\" (UniqueName: \"kubernetes.io/projected/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-kube-api-access-bmngj\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:43.168714 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:43.168486    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:43.168714 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:43.168501    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:43.168714 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:43.168517    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:43.168714 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:43.168530    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:43.168714 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:43.168545    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:43.168714 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:43.168560    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:43.245316 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:43.245278    5315 generic.go:356] "Generic (PLEG): container finished" podID="6ebdce76-7e0e-4916-afcd-2aaf7cc44fed" containerID="7e1c5b05b3783b10ae91334682b73f908c0d459e6b3f486b75b5f62a1c055778" exitCode=0
Nov 03 18:18:43.245493 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:43.245323    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx"
Nov 03 18:18:43.245493 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:43.245358    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx" event={"ID":"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed","Type":"ContainerDied","Data":"7e1c5b05b3783b10ae91334682b73f908c0d459e6b3f486b75b5f62a1c055778"}
Nov 03 18:18:43.245493 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:43.245420    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx" event={"ID":"6ebdce76-7e0e-4916-afcd-2aaf7cc44fed","Type":"ContainerDied","Data":"5fe42f6aa0859da70b8580025d4f7e029430e6ebd4544f864a0ee66fad7a5332"}
Nov 03 18:18:43.245493 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:43.245443    5315 scope.go:122] "RemoveContainer" containerID="7e1c5b05b3783b10ae91334682b73f908c0d459e6b3f486b75b5f62a1c055778"
Nov 03 18:18:43.254970 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:43.254946    5315 scope.go:122] "RemoveContainer" containerID="6c4ad9e24979e6ced486b18cb044721e965b37d80560cf8ccd1905831306733c"
Nov 03 18:18:43.264300 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:43.264273    5315 scope.go:122] "RemoveContainer" containerID="7e1c5b05b3783b10ae91334682b73f908c0d459e6b3f486b75b5f62a1c055778"
Nov 03 18:18:43.264733 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:18:43.264693    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e1c5b05b3783b10ae91334682b73f908c0d459e6b3f486b75b5f62a1c055778\": container with ID starting with 7e1c5b05b3783b10ae91334682b73f908c0d459e6b3f486b75b5f62a1c055778 not found: ID does not exist" containerID="7e1c5b05b3783b10ae91334682b73f908c0d459e6b3f486b75b5f62a1c055778"
Nov 03 18:18:43.264840 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:43.264746    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e1c5b05b3783b10ae91334682b73f908c0d459e6b3f486b75b5f62a1c055778"} err="failed to get container status \"7e1c5b05b3783b10ae91334682b73f908c0d459e6b3f486b75b5f62a1c055778\": rpc error: code = NotFound desc = could not find container \"7e1c5b05b3783b10ae91334682b73f908c0d459e6b3f486b75b5f62a1c055778\": container with ID starting with 7e1c5b05b3783b10ae91334682b73f908c0d459e6b3f486b75b5f62a1c055778 not found: ID does not exist"
Nov 03 18:18:43.264840 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:43.264778    5315 scope.go:122] "RemoveContainer" containerID="6c4ad9e24979e6ced486b18cb044721e965b37d80560cf8ccd1905831306733c"
Nov 03 18:18:43.265130 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:18:43.265078    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c4ad9e24979e6ced486b18cb044721e965b37d80560cf8ccd1905831306733c\": container with ID starting with 6c4ad9e24979e6ced486b18cb044721e965b37d80560cf8ccd1905831306733c not found: ID does not exist" containerID="6c4ad9e24979e6ced486b18cb044721e965b37d80560cf8ccd1905831306733c"
Nov 03 18:18:43.265130 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:43.265112    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c4ad9e24979e6ced486b18cb044721e965b37d80560cf8ccd1905831306733c"} err="failed to get container status \"6c4ad9e24979e6ced486b18cb044721e965b37d80560cf8ccd1905831306733c\": rpc error: code = NotFound desc = could not find container \"6c4ad9e24979e6ced486b18cb044721e965b37d80560cf8ccd1905831306733c\": container with ID starting with 6c4ad9e24979e6ced486b18cb044721e965b37d80560cf8ccd1905831306733c not found: ID does not exist"
Nov 03 18:18:43.404312 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:43.404219    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx"]
Nov 03 18:18:43.408276 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:43.408247    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-c6bxx"]
Nov 03 18:18:45.109102 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:45.109064    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ebdce76-7e0e-4916-afcd-2aaf7cc44fed" path="/var/lib/kubelet/pods/6ebdce76-7e0e-4916-afcd-2aaf7cc44fed/volumes"
Nov 03 18:18:54.577382 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:18:54.577342    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:18:56.045123 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:56.045082    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfefd039-svlhc"]
Nov 03 18:18:56.091157 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:56.091118    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfefd039-svlhc"]
Nov 03 18:18:56.092493 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:56.092466    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f52dedf-zlhqr"]
Nov 03 18:18:56.093585 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:56.093565    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f52dedf-zlhqr"]
Nov 03 18:18:57.103003 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:57.102967    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d467951-e1e8-4900-8d5f-ba26ee79bdd6" path="/var/lib/kubelet/pods/6d467951-e1e8-4900-8d5f-ba26ee79bdd6/volumes"
Nov 03 18:18:57.103412 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:57.103371    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afa2c09e-863f-4086-b3bb-990dc6fe9b94" path="/var/lib/kubelet/pods/afa2c09e-863f-4086-b3bb-990dc6fe9b94/volumes"
Nov 03 18:18:59.808113 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808078    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-nd6sq"]
Nov 03 18:18:59.808641 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808410    5315 cpu_manager.go:414] "containerMap: removing container" podUID="3cf05bbd-6605-4cd2-aa94-85cbb46223aa" containerName="istio-proxy"
Nov 03 18:18:59.808641 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808422    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cf05bbd-6605-4cd2-aa94-85cbb46223aa" containerName="istio-proxy"
Nov 03 18:18:59.808641 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808436    5315 cpu_manager.go:414] "containerMap: removing container" podUID="08159a02-fc2e-4138-824d-be5ba3e141d8" containerName="main"
Nov 03 18:18:59.808641 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808441    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08159a02-fc2e-4138-824d-be5ba3e141d8" containerName="main"
Nov 03 18:18:59.808641 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808448    5315 cpu_manager.go:414] "containerMap: removing container" podUID="6ebdce76-7e0e-4916-afcd-2aaf7cc44fed" containerName="istio-proxy"
Nov 03 18:18:59.808641 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808452    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ebdce76-7e0e-4916-afcd-2aaf7cc44fed" containerName="istio-proxy"
Nov 03 18:18:59.808641 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808459    5315 cpu_manager.go:414] "containerMap: removing container" podUID="cb78e2af-5c83-4d6d-a562-274b12209c20" containerName="istio-proxy"
Nov 03 18:18:59.808641 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808465    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb78e2af-5c83-4d6d-a562-274b12209c20" containerName="istio-proxy"
Nov 03 18:18:59.808641 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808472    5315 cpu_manager.go:414] "containerMap: removing container" podUID="6ebdce76-7e0e-4916-afcd-2aaf7cc44fed" containerName="istio-proxy"
Nov 03 18:18:59.808641 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808476    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ebdce76-7e0e-4916-afcd-2aaf7cc44fed" containerName="istio-proxy"
Nov 03 18:18:59.808641 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808484    5315 cpu_manager.go:414] "containerMap: removing container" podUID="2c20ab79-c080-49c3-a84e-f8f3fc0713f2" containerName="payload-processing"
Nov 03 18:18:59.808641 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808488    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c20ab79-c080-49c3-a84e-f8f3fc0713f2" containerName="payload-processing"
Nov 03 18:18:59.808641 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808497    5315 cpu_manager.go:414] "containerMap: removing container" podUID="16411400-4aa7-4754-9dc2-82175924011a" containerName="istio-proxy"
Nov 03 18:18:59.808641 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808502    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16411400-4aa7-4754-9dc2-82175924011a" containerName="istio-proxy"
Nov 03 18:18:59.808641 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808509    5315 cpu_manager.go:414] "containerMap: removing container" podUID="114dcf54-e4eb-4294-a77b-604fe6d57d48" containerName="main"
Nov 03 18:18:59.808641 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808513    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="114dcf54-e4eb-4294-a77b-604fe6d57d48" containerName="main"
Nov 03 18:18:59.808641 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808521    5315 cpu_manager.go:414] "containerMap: removing container" podUID="cb937207-ced4-4180-a554-0b7a86edce58" containerName="istio-proxy"
Nov 03 18:18:59.808641 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808525    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb937207-ced4-4180-a554-0b7a86edce58" containerName="istio-proxy"
Nov 03 18:18:59.808641 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808533    5315 cpu_manager.go:414] "containerMap: removing container" podUID="6511822b-0252-4d76-88c0-b4e27a0d2e6e" containerName="istio-proxy"
Nov 03 18:18:59.808641 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808537    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6511822b-0252-4d76-88c0-b4e27a0d2e6e" containerName="istio-proxy"
Nov 03 18:18:59.808641 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808545    5315 cpu_manager.go:414] "containerMap: removing container" podUID="91b95076-e971-49d4-b183-66e0da518d56" containerName="revoke-keys"
Nov 03 18:18:59.808641 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808551    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91b95076-e971-49d4-b183-66e0da518d56" containerName="revoke-keys"
Nov 03 18:18:59.808641 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808559    5315 cpu_manager.go:414] "containerMap: removing container" podUID="8b7d3471-e6a0-4514-bf01-afb303808e53" containerName="main"
Nov 03 18:18:59.808641 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808563    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b7d3471-e6a0-4514-bf01-afb303808e53" containerName="main"
Nov 03 18:18:59.808641 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808571    5315 cpu_manager.go:414] "containerMap: removing container" podUID="e376e147-b415-47d0-8bb8-21aecf441da9" containerName="revoke-keys"
Nov 03 18:18:59.808641 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808576    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e376e147-b415-47d0-8bb8-21aecf441da9" containerName="revoke-keys"
Nov 03 18:18:59.808641 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808581    5315 cpu_manager.go:414] "containerMap: removing container" podUID="dafbc7c7-538e-4bf6-ab30-970870f21854" containerName="istio-proxy"
Nov 03 18:18:59.808641 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808585    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dafbc7c7-538e-4bf6-ab30-970870f21854" containerName="istio-proxy"
Nov 03 18:18:59.808641 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808593    5315 cpu_manager.go:414] "containerMap: removing container" podUID="d3aa0994-51ec-4076-bc86-c529ad0bb993" containerName="istio-proxy"
Nov 03 18:18:59.808641 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808598    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3aa0994-51ec-4076-bc86-c529ad0bb993" containerName="istio-proxy"
Nov 03 18:18:59.808641 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808604    5315 cpu_manager.go:414] "containerMap: removing container" podUID="fc4effb6-ba04-4eab-aeaf-a80ef45d2acb" containerName="payload-pre-processing"
Nov 03 18:18:59.808641 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808621    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc4effb6-ba04-4eab-aeaf-a80ef45d2acb" containerName="payload-pre-processing"
Nov 03 18:18:59.808641 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808628    5315 cpu_manager.go:414] "containerMap: removing container" podUID="205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc" containerName="istio-proxy"
Nov 03 18:18:59.808641 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808632    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc" containerName="istio-proxy"
Nov 03 18:18:59.809690 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808684    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="dafbc7c7-538e-4bf6-ab30-970870f21854" containerName="istio-proxy"
Nov 03 18:18:59.809690 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808692    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3aa0994-51ec-4076-bc86-c529ad0bb993" containerName="istio-proxy"
Nov 03 18:18:59.809690 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808698    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc4effb6-ba04-4eab-aeaf-a80ef45d2acb" containerName="payload-pre-processing"
Nov 03 18:18:59.809690 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808704    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="205e40fe-2a6e-4e87-aad5-a2ee6b2dfbcc" containerName="istio-proxy"
Nov 03 18:18:59.809690 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808711    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb78e2af-5c83-4d6d-a562-274b12209c20" containerName="istio-proxy"
Nov 03 18:18:59.809690 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808716    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cf05bbd-6605-4cd2-aa94-85cbb46223aa" containerName="istio-proxy"
Nov 03 18:18:59.809690 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808721    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="08159a02-fc2e-4138-824d-be5ba3e141d8" containerName="main"
Nov 03 18:18:59.809690 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808726    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ebdce76-7e0e-4916-afcd-2aaf7cc44fed" containerName="istio-proxy"
Nov 03 18:18:59.809690 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808730    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb78e2af-5c83-4d6d-a562-274b12209c20" containerName="istio-proxy"
Nov 03 18:18:59.809690 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808734    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ebdce76-7e0e-4916-afcd-2aaf7cc44fed" containerName="istio-proxy"
Nov 03 18:18:59.809690 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808739    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c20ab79-c080-49c3-a84e-f8f3fc0713f2" containerName="payload-processing"
Nov 03 18:18:59.809690 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808745    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="16411400-4aa7-4754-9dc2-82175924011a" containerName="istio-proxy"
Nov 03 18:18:59.809690 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808750    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="114dcf54-e4eb-4294-a77b-604fe6d57d48" containerName="main"
Nov 03 18:18:59.809690 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808755    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb937207-ced4-4180-a554-0b7a86edce58" containerName="istio-proxy"
Nov 03 18:18:59.809690 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808760    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="6511822b-0252-4d76-88c0-b4e27a0d2e6e" containerName="istio-proxy"
Nov 03 18:18:59.809690 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808765    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="91b95076-e971-49d4-b183-66e0da518d56" containerName="revoke-keys"
Nov 03 18:18:59.809690 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808769    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b7d3471-e6a0-4514-bf01-afb303808e53" containerName="main"
Nov 03 18:18:59.809690 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.808775    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="e376e147-b415-47d0-8bb8-21aecf441da9" containerName="revoke-keys"
Nov 03 18:18:59.812381 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.812360    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-nd6sq"
Nov 03 18:18:59.854153 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.854125    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dzcvs"]
Nov 03 18:18:59.854500 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.854486    5315 cpu_manager.go:414] "containerMap: removing container" podUID="cb78e2af-5c83-4d6d-a562-274b12209c20" containerName="istio-proxy"
Nov 03 18:18:59.854546 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.854504    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb78e2af-5c83-4d6d-a562-274b12209c20" containerName="istio-proxy"
Nov 03 18:18:59.857516 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.857501    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7jkn9"]
Nov 03 18:18:59.857657 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.857635    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dzcvs"
Nov 03 18:18:59.860627 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.860582    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7jkn9"
Nov 03 18:18:59.868795 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.868773    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4jx7c"]
Nov 03 18:18:59.872013 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.871996    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4jx7c"
Nov 03 18:18:59.898770 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.898748    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-h54vm\""
Nov 03 18:18:59.922890 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.922859    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-nd6sq"]
Nov 03 18:18:59.942323 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.942284    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-4jx7c\" (UID: \"8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4jx7c"
Nov 03 18:18:59.942323 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.942326    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e4a915b-dd46-4d77-b425-8c37440fdaad-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-dzcvs\" (UID: \"7e4a915b-dd46-4d77-b425-8c37440fdaad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dzcvs"
Nov 03 18:18:59.942569 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.942403    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cccc85e9-7f5c-4a60-a245-dacb32d91965-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-nd6sq\" (UID: \"cccc85e9-7f5c-4a60-a245-dacb32d91965\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-nd6sq"
Nov 03 18:18:59.942569 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.942438    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e37d948-2760-4441-8d44-722000f7e5b2-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-7jkn9\" (UID: \"0e37d948-2760-4441-8d44-722000f7e5b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7jkn9"
Nov 03 18:18:59.942569 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.942460    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cccc85e9-7f5c-4a60-a245-dacb32d91965-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-nd6sq\" (UID: \"cccc85e9-7f5c-4a60-a245-dacb32d91965\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-nd6sq"
Nov 03 18:18:59.942569 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.942547    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e37d948-2760-4441-8d44-722000f7e5b2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-7jkn9\" (UID: \"0e37d948-2760-4441-8d44-722000f7e5b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7jkn9"
Nov 03 18:18:59.942839 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.942574    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x4w5\" (UniqueName: \"kubernetes.io/projected/0e37d948-2760-4441-8d44-722000f7e5b2-kube-api-access-7x4w5\") pod \"maas-default-gateway-openshift-default-6dcf757899-7jkn9\" (UID: \"0e37d948-2760-4441-8d44-722000f7e5b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7jkn9"
Nov 03 18:18:59.942839 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.942595    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e37d948-2760-4441-8d44-722000f7e5b2-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-7jkn9\" (UID: \"0e37d948-2760-4441-8d44-722000f7e5b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7jkn9"
Nov 03 18:18:59.942839 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.942631    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cccc85e9-7f5c-4a60-a245-dacb32d91965-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-nd6sq\" (UID: \"cccc85e9-7f5c-4a60-a245-dacb32d91965\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-nd6sq"
Nov 03 18:18:59.942839 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.942683    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e37d948-2760-4441-8d44-722000f7e5b2-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-7jkn9\" (UID: \"0e37d948-2760-4441-8d44-722000f7e5b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7jkn9"
Nov 03 18:18:59.942839 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.942716    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-4jx7c\" (UID: \"8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4jx7c"
Nov 03 18:18:59.942839 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.942743    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e37d948-2760-4441-8d44-722000f7e5b2-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-7jkn9\" (UID: \"0e37d948-2760-4441-8d44-722000f7e5b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7jkn9"
Nov 03 18:18:59.942839 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.942762    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e4a915b-dd46-4d77-b425-8c37440fdaad-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-dzcvs\" (UID: \"7e4a915b-dd46-4d77-b425-8c37440fdaad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dzcvs"
Nov 03 18:18:59.943296 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.942846    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e4a915b-dd46-4d77-b425-8c37440fdaad-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-dzcvs\" (UID: \"7e4a915b-dd46-4d77-b425-8c37440fdaad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dzcvs"
Nov 03 18:18:59.943296 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.942864    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cccc85e9-7f5c-4a60-a245-dacb32d91965-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-nd6sq\" (UID: \"cccc85e9-7f5c-4a60-a245-dacb32d91965\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-nd6sq"
Nov 03 18:18:59.943296 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.942883    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-4jx7c\" (UID: \"8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4jx7c"
Nov 03 18:18:59.943296 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.942903    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-4jx7c\" (UID: \"8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4jx7c"
Nov 03 18:18:59.943296 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.942950    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-4jx7c\" (UID: \"8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4jx7c"
Nov 03 18:18:59.943296 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.942993    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-4jx7c\" (UID: \"8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4jx7c"
Nov 03 18:18:59.943296 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.943044    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-4jx7c\" (UID: \"8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4jx7c"
Nov 03 18:18:59.943296 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.943067    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkmpg\" (UniqueName: \"kubernetes.io/projected/7e4a915b-dd46-4d77-b425-8c37440fdaad-kube-api-access-pkmpg\") pod \"maas-default-gateway-openshift-default-6dcf757899-dzcvs\" (UID: \"7e4a915b-dd46-4d77-b425-8c37440fdaad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dzcvs"
Nov 03 18:18:59.943296 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.943090    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cccc85e9-7f5c-4a60-a245-dacb32d91965-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-nd6sq\" (UID: \"cccc85e9-7f5c-4a60-a245-dacb32d91965\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-nd6sq"
Nov 03 18:18:59.943296 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.943114    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e4a915b-dd46-4d77-b425-8c37440fdaad-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-dzcvs\" (UID: \"7e4a915b-dd46-4d77-b425-8c37440fdaad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dzcvs"
Nov 03 18:18:59.943296 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.943134    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cccc85e9-7f5c-4a60-a245-dacb32d91965-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-nd6sq\" (UID: \"cccc85e9-7f5c-4a60-a245-dacb32d91965\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-nd6sq"
Nov 03 18:18:59.943296 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.943159    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jndqp\" (UniqueName: \"kubernetes.io/projected/cccc85e9-7f5c-4a60-a245-dacb32d91965-kube-api-access-jndqp\") pod \"maas-default-gateway-openshift-default-6dcf757899-nd6sq\" (UID: \"cccc85e9-7f5c-4a60-a245-dacb32d91965\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-nd6sq"
Nov 03 18:18:59.943296 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.943180    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e4a915b-dd46-4d77-b425-8c37440fdaad-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-dzcvs\" (UID: \"7e4a915b-dd46-4d77-b425-8c37440fdaad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dzcvs"
Nov 03 18:18:59.943296 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.943221    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cccc85e9-7f5c-4a60-a245-dacb32d91965-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-nd6sq\" (UID: \"cccc85e9-7f5c-4a60-a245-dacb32d91965\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-nd6sq"
Nov 03 18:18:59.943296 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.943264    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e37d948-2760-4441-8d44-722000f7e5b2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-7jkn9\" (UID: \"0e37d948-2760-4441-8d44-722000f7e5b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7jkn9"
Nov 03 18:18:59.943296 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.943283    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e4a915b-dd46-4d77-b425-8c37440fdaad-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-dzcvs\" (UID: \"7e4a915b-dd46-4d77-b425-8c37440fdaad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dzcvs"
Nov 03 18:18:59.943296 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.943305    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e37d948-2760-4441-8d44-722000f7e5b2-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-7jkn9\" (UID: \"0e37d948-2760-4441-8d44-722000f7e5b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7jkn9"
Nov 03 18:18:59.944272 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.943359    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e4a915b-dd46-4d77-b425-8c37440fdaad-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-dzcvs\" (UID: \"7e4a915b-dd46-4d77-b425-8c37440fdaad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dzcvs"
Nov 03 18:18:59.944272 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.943378    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e4a915b-dd46-4d77-b425-8c37440fdaad-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-dzcvs\" (UID: \"7e4a915b-dd46-4d77-b425-8c37440fdaad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dzcvs"
Nov 03 18:18:59.944272 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.943403    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e37d948-2760-4441-8d44-722000f7e5b2-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-7jkn9\" (UID: \"0e37d948-2760-4441-8d44-722000f7e5b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7jkn9"
Nov 03 18:18:59.944272 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.943448    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r69sq\" (UniqueName: \"kubernetes.io/projected/8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8-kube-api-access-r69sq\") pod \"maas-default-gateway-openshift-default-6dcf757899-4jx7c\" (UID: \"8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4jx7c"
Nov 03 18:18:59.944272 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.943475    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-4jx7c\" (UID: \"8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4jx7c"
Nov 03 18:18:59.944272 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.943495    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cccc85e9-7f5c-4a60-a245-dacb32d91965-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-nd6sq\" (UID: \"cccc85e9-7f5c-4a60-a245-dacb32d91965\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-nd6sq"
Nov 03 18:18:59.976305 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.976269    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dzcvs"]
Nov 03 18:18:59.977779 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.977752    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7jkn9"]
Nov 03 18:18:59.979282 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:18:59.979259    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4jx7c"]
Nov 03 18:19:00.044048 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.044009    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e4a915b-dd46-4d77-b425-8c37440fdaad-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-dzcvs\" (UID: \"7e4a915b-dd46-4d77-b425-8c37440fdaad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dzcvs"
Nov 03 18:19:00.044048 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.044047    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e4a915b-dd46-4d77-b425-8c37440fdaad-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-dzcvs\" (UID: \"7e4a915b-dd46-4d77-b425-8c37440fdaad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dzcvs"
Nov 03 18:19:00.044284 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.044167    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e37d948-2760-4441-8d44-722000f7e5b2-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-7jkn9\" (UID: \"0e37d948-2760-4441-8d44-722000f7e5b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7jkn9"
Nov 03 18:19:00.044284 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.044204    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r69sq\" (UniqueName: \"kubernetes.io/projected/8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8-kube-api-access-r69sq\") pod \"maas-default-gateway-openshift-default-6dcf757899-4jx7c\" (UID: \"8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4jx7c"
Nov 03 18:19:00.044284 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.044240    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-4jx7c\" (UID: \"8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4jx7c"
Nov 03 18:19:00.044284 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.044278    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cccc85e9-7f5c-4a60-a245-dacb32d91965-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-nd6sq\" (UID: \"cccc85e9-7f5c-4a60-a245-dacb32d91965\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-nd6sq"
Nov 03 18:19:00.044491 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.044308    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-4jx7c\" (UID: \"8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4jx7c"
Nov 03 18:19:00.044491 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.044366    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e4a915b-dd46-4d77-b425-8c37440fdaad-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-dzcvs\" (UID: \"7e4a915b-dd46-4d77-b425-8c37440fdaad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dzcvs"
Nov 03 18:19:00.044491 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.044418    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cccc85e9-7f5c-4a60-a245-dacb32d91965-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-nd6sq\" (UID: \"cccc85e9-7f5c-4a60-a245-dacb32d91965\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-nd6sq"
Nov 03 18:19:00.044491 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.044452    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e37d948-2760-4441-8d44-722000f7e5b2-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-7jkn9\" (UID: \"0e37d948-2760-4441-8d44-722000f7e5b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7jkn9"
Nov 03 18:19:00.044491 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.044476    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cccc85e9-7f5c-4a60-a245-dacb32d91965-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-nd6sq\" (UID: \"cccc85e9-7f5c-4a60-a245-dacb32d91965\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-nd6sq"
Nov 03 18:19:00.044784 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.044507    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e37d948-2760-4441-8d44-722000f7e5b2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-7jkn9\" (UID: \"0e37d948-2760-4441-8d44-722000f7e5b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7jkn9"
Nov 03 18:19:00.044784 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.044538    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x4w5\" (UniqueName: \"kubernetes.io/projected/0e37d948-2760-4441-8d44-722000f7e5b2-kube-api-access-7x4w5\") pod \"maas-default-gateway-openshift-default-6dcf757899-7jkn9\" (UID: \"0e37d948-2760-4441-8d44-722000f7e5b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7jkn9"
Nov 03 18:19:00.044784 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.044569    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e37d948-2760-4441-8d44-722000f7e5b2-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-7jkn9\" (UID: \"0e37d948-2760-4441-8d44-722000f7e5b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7jkn9"
Nov 03 18:19:00.044784 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.044581    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e37d948-2760-4441-8d44-722000f7e5b2-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-7jkn9\" (UID: \"0e37d948-2760-4441-8d44-722000f7e5b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7jkn9"
Nov 03 18:19:00.044784 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.044657    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-4jx7c\" (UID: \"8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4jx7c"
Nov 03 18:19:00.045062 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.044857    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e37d948-2760-4441-8d44-722000f7e5b2-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-7jkn9\" (UID: \"0e37d948-2760-4441-8d44-722000f7e5b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7jkn9"
Nov 03 18:19:00.045062 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.044594    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cccc85e9-7f5c-4a60-a245-dacb32d91965-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-nd6sq\" (UID: \"cccc85e9-7f5c-4a60-a245-dacb32d91965\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-nd6sq"
Nov 03 18:19:00.045062 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.044914    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e4a915b-dd46-4d77-b425-8c37440fdaad-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-dzcvs\" (UID: \"7e4a915b-dd46-4d77-b425-8c37440fdaad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dzcvs"
Nov 03 18:19:00.045062 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.044921    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e37d948-2760-4441-8d44-722000f7e5b2-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-7jkn9\" (UID: \"0e37d948-2760-4441-8d44-722000f7e5b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7jkn9"
Nov 03 18:19:00.045062 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.044964    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cccc85e9-7f5c-4a60-a245-dacb32d91965-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-nd6sq\" (UID: \"cccc85e9-7f5c-4a60-a245-dacb32d91965\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-nd6sq"
Nov 03 18:19:00.045062 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.044969    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e4a915b-dd46-4d77-b425-8c37440fdaad-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-dzcvs\" (UID: \"7e4a915b-dd46-4d77-b425-8c37440fdaad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dzcvs"
Nov 03 18:19:00.045062 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.044976    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-4jx7c\" (UID: \"8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4jx7c"
Nov 03 18:19:00.045062 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.045035    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e37d948-2760-4441-8d44-722000f7e5b2-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-7jkn9\" (UID: \"0e37d948-2760-4441-8d44-722000f7e5b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7jkn9"
Nov 03 18:19:00.045509 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.045066    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e4a915b-dd46-4d77-b425-8c37440fdaad-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-dzcvs\" (UID: \"7e4a915b-dd46-4d77-b425-8c37440fdaad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dzcvs"
Nov 03 18:19:00.045509 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.045111    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e4a915b-dd46-4d77-b425-8c37440fdaad-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-dzcvs\" (UID: \"7e4a915b-dd46-4d77-b425-8c37440fdaad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dzcvs"
Nov 03 18:19:00.045509 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.045136    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cccc85e9-7f5c-4a60-a245-dacb32d91965-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-nd6sq\" (UID: \"cccc85e9-7f5c-4a60-a245-dacb32d91965\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-nd6sq"
Nov 03 18:19:00.045509 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.045152    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e37d948-2760-4441-8d44-722000f7e5b2-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-7jkn9\" (UID: \"0e37d948-2760-4441-8d44-722000f7e5b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7jkn9"
Nov 03 18:19:00.045509 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.045165    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-4jx7c\" (UID: \"8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4jx7c"
Nov 03 18:19:00.045509 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.045176    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cccc85e9-7f5c-4a60-a245-dacb32d91965-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-nd6sq\" (UID: \"cccc85e9-7f5c-4a60-a245-dacb32d91965\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-nd6sq"
Nov 03 18:19:00.045509 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.045206    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-4jx7c\" (UID: \"8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4jx7c"
Nov 03 18:19:00.045509 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.045232    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-4jx7c\" (UID: \"8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4jx7c"
Nov 03 18:19:00.045509 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.045239    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-4jx7c\" (UID: \"8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4jx7c"
Nov 03 18:19:00.045509 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.045265    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-4jx7c\" (UID: \"8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4jx7c"
Nov 03 18:19:00.045509 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.045304    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-4jx7c\" (UID: \"8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4jx7c"
Nov 03 18:19:00.045509 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.045342    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkmpg\" (UniqueName: \"kubernetes.io/projected/7e4a915b-dd46-4d77-b425-8c37440fdaad-kube-api-access-pkmpg\") pod \"maas-default-gateway-openshift-default-6dcf757899-dzcvs\" (UID: \"7e4a915b-dd46-4d77-b425-8c37440fdaad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dzcvs"
Nov 03 18:19:00.045509 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.045382    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cccc85e9-7f5c-4a60-a245-dacb32d91965-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-nd6sq\" (UID: \"cccc85e9-7f5c-4a60-a245-dacb32d91965\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-nd6sq"
Nov 03 18:19:00.045509 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.045411    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e4a915b-dd46-4d77-b425-8c37440fdaad-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-dzcvs\" (UID: \"7e4a915b-dd46-4d77-b425-8c37440fdaad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dzcvs"
Nov 03 18:19:00.045509 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.045423    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-4jx7c\" (UID: \"8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4jx7c"
Nov 03 18:19:00.045509 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.045437    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cccc85e9-7f5c-4a60-a245-dacb32d91965-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-nd6sq\" (UID: \"cccc85e9-7f5c-4a60-a245-dacb32d91965\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-nd6sq"
Nov 03 18:19:00.045509 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.045465    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jndqp\" (UniqueName: \"kubernetes.io/projected/cccc85e9-7f5c-4a60-a245-dacb32d91965-kube-api-access-jndqp\") pod \"maas-default-gateway-openshift-default-6dcf757899-nd6sq\" (UID: \"cccc85e9-7f5c-4a60-a245-dacb32d91965\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-nd6sq"
Nov 03 18:19:00.045509 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.045491    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e4a915b-dd46-4d77-b425-8c37440fdaad-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-dzcvs\" (UID: \"7e4a915b-dd46-4d77-b425-8c37440fdaad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dzcvs"
Nov 03 18:19:00.045509 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.045519    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cccc85e9-7f5c-4a60-a245-dacb32d91965-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-nd6sq\" (UID: \"cccc85e9-7f5c-4a60-a245-dacb32d91965\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-nd6sq"
Nov 03 18:19:00.046638 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.045550    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e37d948-2760-4441-8d44-722000f7e5b2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-7jkn9\" (UID: \"0e37d948-2760-4441-8d44-722000f7e5b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7jkn9"
Nov 03 18:19:00.046638 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.045580    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e4a915b-dd46-4d77-b425-8c37440fdaad-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-dzcvs\" (UID: \"7e4a915b-dd46-4d77-b425-8c37440fdaad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dzcvs"
Nov 03 18:19:00.046638 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.045605    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e37d948-2760-4441-8d44-722000f7e5b2-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-7jkn9\" (UID: \"0e37d948-2760-4441-8d44-722000f7e5b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7jkn9"
Nov 03 18:19:00.046638 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.045693    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e37d948-2760-4441-8d44-722000f7e5b2-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-7jkn9\" (UID: \"0e37d948-2760-4441-8d44-722000f7e5b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7jkn9"
Nov 03 18:19:00.046638 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.046158    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-4jx7c\" (UID: \"8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4jx7c"
Nov 03 18:19:00.046944 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.046782    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e37d948-2760-4441-8d44-722000f7e5b2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-7jkn9\" (UID: \"0e37d948-2760-4441-8d44-722000f7e5b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7jkn9"
Nov 03 18:19:00.046944 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.046821    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cccc85e9-7f5c-4a60-a245-dacb32d91965-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-nd6sq\" (UID: \"cccc85e9-7f5c-4a60-a245-dacb32d91965\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-nd6sq"
Nov 03 18:19:00.046944 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.046882    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cccc85e9-7f5c-4a60-a245-dacb32d91965-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-nd6sq\" (UID: \"cccc85e9-7f5c-4a60-a245-dacb32d91965\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-nd6sq"
Nov 03 18:19:00.047122 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.047008    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-4jx7c\" (UID: \"8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4jx7c"
Nov 03 18:19:00.047122 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.047103    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e4a915b-dd46-4d77-b425-8c37440fdaad-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-dzcvs\" (UID: \"7e4a915b-dd46-4d77-b425-8c37440fdaad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dzcvs"
Nov 03 18:19:00.047600 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.047254    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e4a915b-dd46-4d77-b425-8c37440fdaad-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-dzcvs\" (UID: \"7e4a915b-dd46-4d77-b425-8c37440fdaad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dzcvs"
Nov 03 18:19:00.047600 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.047347    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cccc85e9-7f5c-4a60-a245-dacb32d91965-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-nd6sq\" (UID: \"cccc85e9-7f5c-4a60-a245-dacb32d91965\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-nd6sq"
Nov 03 18:19:00.047600 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.047551    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e4a915b-dd46-4d77-b425-8c37440fdaad-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-dzcvs\" (UID: \"7e4a915b-dd46-4d77-b425-8c37440fdaad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dzcvs"
Nov 03 18:19:00.048111 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.048083    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-4jx7c\" (UID: \"8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4jx7c"
Nov 03 18:19:00.048291 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.048269    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e4a915b-dd46-4d77-b425-8c37440fdaad-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-dzcvs\" (UID: \"7e4a915b-dd46-4d77-b425-8c37440fdaad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dzcvs"
Nov 03 18:19:00.048291 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.048280    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e4a915b-dd46-4d77-b425-8c37440fdaad-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-dzcvs\" (UID: \"7e4a915b-dd46-4d77-b425-8c37440fdaad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dzcvs"
Nov 03 18:19:00.048508 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.048484    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cccc85e9-7f5c-4a60-a245-dacb32d91965-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-nd6sq\" (UID: \"cccc85e9-7f5c-4a60-a245-dacb32d91965\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-nd6sq"
Nov 03 18:19:00.048958 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.048936    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e37d948-2760-4441-8d44-722000f7e5b2-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-7jkn9\" (UID: \"0e37d948-2760-4441-8d44-722000f7e5b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7jkn9"
Nov 03 18:19:00.049149 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.049127    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-4jx7c\" (UID: \"8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4jx7c"
Nov 03 18:19:00.049325 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.049304    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cccc85e9-7f5c-4a60-a245-dacb32d91965-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-nd6sq\" (UID: \"cccc85e9-7f5c-4a60-a245-dacb32d91965\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-nd6sq"
Nov 03 18:19:00.049941 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.049921    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e37d948-2760-4441-8d44-722000f7e5b2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-7jkn9\" (UID: \"0e37d948-2760-4441-8d44-722000f7e5b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7jkn9"
Nov 03 18:19:00.102896 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.102819    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x4w5\" (UniqueName: \"kubernetes.io/projected/0e37d948-2760-4441-8d44-722000f7e5b2-kube-api-access-7x4w5\") pod \"maas-default-gateway-openshift-default-6dcf757899-7jkn9\" (UID: \"0e37d948-2760-4441-8d44-722000f7e5b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7jkn9"
Nov 03 18:19:00.102896 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.102859    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e37d948-2760-4441-8d44-722000f7e5b2-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-7jkn9\" (UID: \"0e37d948-2760-4441-8d44-722000f7e5b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7jkn9"
Nov 03 18:19:00.102896 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.102830    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cccc85e9-7f5c-4a60-a245-dacb32d91965-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-nd6sq\" (UID: \"cccc85e9-7f5c-4a60-a245-dacb32d91965\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-nd6sq"
Nov 03 18:19:00.102896 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.102857    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r69sq\" (UniqueName: \"kubernetes.io/projected/8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8-kube-api-access-r69sq\") pod \"maas-default-gateway-openshift-default-6dcf757899-4jx7c\" (UID: \"8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4jx7c"
Nov 03 18:19:00.103135 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.102897    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jndqp\" (UniqueName: \"kubernetes.io/projected/cccc85e9-7f5c-4a60-a245-dacb32d91965-kube-api-access-jndqp\") pod \"maas-default-gateway-openshift-default-6dcf757899-nd6sq\" (UID: \"cccc85e9-7f5c-4a60-a245-dacb32d91965\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-nd6sq"
Nov 03 18:19:00.103324 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.103306    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkmpg\" (UniqueName: \"kubernetes.io/projected/7e4a915b-dd46-4d77-b425-8c37440fdaad-kube-api-access-pkmpg\") pod \"maas-default-gateway-openshift-default-6dcf757899-dzcvs\" (UID: \"7e4a915b-dd46-4d77-b425-8c37440fdaad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dzcvs"
Nov 03 18:19:00.105830 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.105811    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-4jx7c\" (UID: \"8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4jx7c"
Nov 03 18:19:00.110120 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.110095    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e4a915b-dd46-4d77-b425-8c37440fdaad-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-dzcvs\" (UID: \"7e4a915b-dd46-4d77-b425-8c37440fdaad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dzcvs"
Nov 03 18:19:00.124836 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.124814    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-nd6sq"
Nov 03 18:19:00.171037 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.171002    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dzcvs"
Nov 03 18:19:00.178846 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.178816    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7jkn9"
Nov 03 18:19:00.189304 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.187563    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4jx7c"
Nov 03 18:19:00.549091 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.549052    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-nd6sq"]
Nov 03 18:19:00.555461 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:19:00.555427    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcccc85e9_7f5c_4a60_a245_dacb32d91965.slice/crio-d8f71d0b19bbd5ee79de00d88192f87f131e20c84f2d43a6c9a250305bff4e41 WatchSource:0}: Error finding container d8f71d0b19bbd5ee79de00d88192f87f131e20c84f2d43a6c9a250305bff4e41: Status 404 returned error can't find the container with id d8f71d0b19bbd5ee79de00d88192f87f131e20c84f2d43a6c9a250305bff4e41
Nov 03 18:19:00.557772 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.557739    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:19:00.557880 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.557807    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:19:00.557880 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.557838    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:19:00.602588 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.602563    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4jx7c"]
Nov 03 18:19:00.609025 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:19:00.608995    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c2cfd87_4fa6_4780_8d48_4f93dd5d1dc8.slice/crio-4365b55c37abffed65036a34ff75d1ffbf888cc070834dd7ba2184a2baf78b8f WatchSource:0}: Error finding container 4365b55c37abffed65036a34ff75d1ffbf888cc070834dd7ba2184a2baf78b8f: Status 404 returned error can't find the container with id 4365b55c37abffed65036a34ff75d1ffbf888cc070834dd7ba2184a2baf78b8f
Nov 03 18:19:00.611395 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.611363    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:19:00.611504 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.611428    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:19:00.611504 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.611458    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:19:00.652446 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.652366    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7jkn9"]
Nov 03 18:19:00.654841 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.654814    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dzcvs"]
Nov 03 18:19:00.659110 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:19:00.659070    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e37d948_2760_4441_8d44_722000f7e5b2.slice/crio-04da6373770d4d762d36cd8b9a2c1736dc6ca41a39fd3847ea20da0ceb67e041 WatchSource:0}: Error finding container 04da6373770d4d762d36cd8b9a2c1736dc6ca41a39fd3847ea20da0ceb67e041: Status 404 returned error can't find the container with id 04da6373770d4d762d36cd8b9a2c1736dc6ca41a39fd3847ea20da0ceb67e041
Nov 03 18:19:00.661987 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.661946    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:19:00.662073 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.662026    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:19:00.662073 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.662056    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:19:00.672064 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:19:00.672033    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e4a915b_dd46_4d77_b425_8c37440fdaad.slice/crio-aba539ea8fd4059958f8366620760fb1fa0c7d9523e40f8d4c0515309b458b1f WatchSource:0}: Error finding container aba539ea8fd4059958f8366620760fb1fa0c7d9523e40f8d4c0515309b458b1f: Status 404 returned error can't find the container with id aba539ea8fd4059958f8366620760fb1fa0c7d9523e40f8d4c0515309b458b1f
Nov 03 18:19:00.675246 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.675216    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:19:00.675310 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.675283    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:19:00.675353 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:00.675311    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:19:01.506339 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:01.506283    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-nd6sq" event={"ID":"cccc85e9-7f5c-4a60-a245-dacb32d91965","Type":"ContainerStarted","Data":"02dca36680754117aee317186f695943726a206a0e8963b917299c1ea6ee8407"}
Nov 03 18:19:01.507041 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:01.507009    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-nd6sq" event={"ID":"cccc85e9-7f5c-4a60-a245-dacb32d91965","Type":"ContainerStarted","Data":"d8f71d0b19bbd5ee79de00d88192f87f131e20c84f2d43a6c9a250305bff4e41"}
Nov 03 18:19:01.511427 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:01.511395    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7jkn9" event={"ID":"0e37d948-2760-4441-8d44-722000f7e5b2","Type":"ContainerStarted","Data":"2742e0995424dd2d64c763a5017d640e25057545cc7522905e889e8148efb7ae"}
Nov 03 18:19:01.511679 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:01.511654    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7jkn9" event={"ID":"0e37d948-2760-4441-8d44-722000f7e5b2","Type":"ContainerStarted","Data":"04da6373770d4d762d36cd8b9a2c1736dc6ca41a39fd3847ea20da0ceb67e041"}
Nov 03 18:19:01.517707 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:01.517672    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4jx7c" event={"ID":"8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8","Type":"ContainerStarted","Data":"f8e1faec6f65807f001e22eb39e628b72805b01598ebf7f36b99aed904349099"}
Nov 03 18:19:01.519293 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:01.519265    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4jx7c" event={"ID":"8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8","Type":"ContainerStarted","Data":"4365b55c37abffed65036a34ff75d1ffbf888cc070834dd7ba2184a2baf78b8f"}
Nov 03 18:19:01.520953 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:01.520925    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dzcvs" event={"ID":"7e4a915b-dd46-4d77-b425-8c37440fdaad","Type":"ContainerStarted","Data":"e2644c827289758a3f379adc99fb7018e7f0b1565f8a30e247a775cff3f16be8"}
Nov 03 18:19:01.521772 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:01.521750    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dzcvs" event={"ID":"7e4a915b-dd46-4d77-b425-8c37440fdaad","Type":"ContainerStarted","Data":"aba539ea8fd4059958f8366620760fb1fa0c7d9523e40f8d4c0515309b458b1f"}
Nov 03 18:19:01.748876 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:01.746519    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-nd6sq" podStartSLOduration=2.74649771 podStartE2EDuration="2.74649771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:18:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:19:01.628945914 +0000 UTC m=+1980.645143997" watchObservedRunningTime="2026-11-03 18:19:01.74649771 +0000 UTC m=+1980.762695780"
Nov 03 18:19:01.879759 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:01.877790    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dzcvs" podStartSLOduration=2.877769642 podStartE2EDuration="2.877769642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:18:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:19:01.875784363 +0000 UTC m=+1980.891982433" watchObservedRunningTime="2026-11-03 18:19:01.877769642 +0000 UTC m=+1980.893967713"
Nov 03 18:19:01.879759 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:01.878282    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4jx7c" podStartSLOduration=2.878271649 podStartE2EDuration="2.878271649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:18:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:19:01.760753712 +0000 UTC m=+1980.776951781" watchObservedRunningTime="2026-11-03 18:19:01.878271649 +0000 UTC m=+1980.894469718"
Nov 03 18:19:01.992634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:01.991688    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7jkn9" podStartSLOduration=2.991667196 podStartE2EDuration="2.991667196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:18:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:19:01.990278485 +0000 UTC m=+1981.006476578" watchObservedRunningTime="2026-11-03 18:19:01.991667196 +0000 UTC m=+1981.007865266"
Nov 03 18:19:02.126634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:02.126091    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-nd6sq"
Nov 03 18:19:02.131690 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:02.130435    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-nd6sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Nov 03 18:19:02.131690 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:02.130527    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-nd6sq" podUID="cccc85e9-7f5c-4a60-a245-dacb32d91965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Nov 03 18:19:02.173634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:02.173501    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dzcvs"
Nov 03 18:19:02.173862 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:02.173717    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-dzcvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Nov 03 18:19:02.173862 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:02.173788    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dzcvs" podUID="7e4a915b-dd46-4d77-b425-8c37440fdaad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Nov 03 18:19:02.182395 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:02.182298    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7jkn9"
Nov 03 18:19:02.184189 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:02.184156    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-7jkn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Nov 03 18:19:02.184463 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:02.184415    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7jkn9" podUID="0e37d948-2760-4441-8d44-722000f7e5b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Nov 03 18:19:02.190577 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:02.188975    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4jx7c"
Nov 03 18:19:02.190577 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:02.190428    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-4jx7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Nov 03 18:19:02.190577 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:02.190500    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4jx7c" podUID="8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Nov 03 18:19:03.129069 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:03.129030    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-nd6sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Nov 03 18:19:03.138190 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:03.137717    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-nd6sq" podUID="cccc85e9-7f5c-4a60-a245-dacb32d91965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Nov 03 18:19:03.173672 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:03.173634    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-dzcvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Nov 03 18:19:03.174398 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:03.174363    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dzcvs" podUID="7e4a915b-dd46-4d77-b425-8c37440fdaad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Nov 03 18:19:03.182138 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:03.182105    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-7jkn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Nov 03 18:19:03.182371 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:03.182338    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7jkn9" podUID="0e37d948-2760-4441-8d44-722000f7e5b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Nov 03 18:19:03.189991 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:03.189960    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-4jx7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Nov 03 18:19:03.190201 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:03.190173    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4jx7c" podUID="8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Nov 03 18:19:04.130337 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:04.130297    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-nd6sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Nov 03 18:19:04.131145 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:04.131113    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-nd6sq" podUID="cccc85e9-7f5c-4a60-a245-dacb32d91965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Nov 03 18:19:04.174100 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:04.174057    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-dzcvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Nov 03 18:19:04.174366 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:04.174335    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dzcvs" podUID="7e4a915b-dd46-4d77-b425-8c37440fdaad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Nov 03 18:19:04.185066 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:04.185027    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-7jkn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Nov 03 18:19:04.185350 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:04.185320    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7jkn9" podUID="0e37d948-2760-4441-8d44-722000f7e5b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Nov 03 18:19:04.190037 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:04.190000    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-4jx7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Nov 03 18:19:04.190276 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:04.190247    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4jx7c" podUID="8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Nov 03 18:19:05.134635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:05.133654    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-nd6sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Nov 03 18:19:05.134635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:05.133720    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-nd6sq" podUID="cccc85e9-7f5c-4a60-a245-dacb32d91965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Nov 03 18:19:05.174854 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:05.174673    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-dzcvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Nov 03 18:19:05.174854 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:05.174772    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dzcvs" podUID="7e4a915b-dd46-4d77-b425-8c37440fdaad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Nov 03 18:19:05.179636 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:05.179490    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-7jkn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Nov 03 18:19:05.179636 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:05.179544    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7jkn9" podUID="0e37d948-2760-4441-8d44-722000f7e5b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Nov 03 18:19:05.202248 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:05.202213    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-4jx7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Nov 03 18:19:05.202476 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:05.202446    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4jx7c" podUID="8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Nov 03 18:19:06.128104 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:06.128057    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-nd6sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Nov 03 18:19:06.128457 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:06.128423    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-nd6sq" podUID="cccc85e9-7f5c-4a60-a245-dacb32d91965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Nov 03 18:19:06.175121 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:06.175077    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-dzcvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Nov 03 18:19:06.175758 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:06.175723    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dzcvs" podUID="7e4a915b-dd46-4d77-b425-8c37440fdaad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Nov 03 18:19:06.184637 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:06.180081    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-7jkn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Nov 03 18:19:06.184637 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:06.180145    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7jkn9" podUID="0e37d948-2760-4441-8d44-722000f7e5b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Nov 03 18:19:06.188996 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:06.188963    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-4jx7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Nov 03 18:19:06.189230 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:06.189200    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4jx7c" podUID="8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Nov 03 18:19:07.126705 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:07.126229    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-nd6sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Nov 03 18:19:07.126705 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:07.126291    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-nd6sq" podUID="cccc85e9-7f5c-4a60-a245-dacb32d91965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Nov 03 18:19:07.175171 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:07.175023    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-dzcvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Nov 03 18:19:07.175171 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:07.175107    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dzcvs" podUID="7e4a915b-dd46-4d77-b425-8c37440fdaad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Nov 03 18:19:07.186696 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:07.186547    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-7jkn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Nov 03 18:19:07.186696 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:07.186633    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7jkn9" podUID="0e37d948-2760-4441-8d44-722000f7e5b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Nov 03 18:19:07.191077 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:07.190963    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-4jx7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Nov 03 18:19:07.191077 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:07.191021    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4jx7c" podUID="8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Nov 03 18:19:08.126070 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:08.126015    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-nd6sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Nov 03 18:19:08.126302 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:08.126103    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-nd6sq" podUID="cccc85e9-7f5c-4a60-a245-dacb32d91965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Nov 03 18:19:08.171540 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:08.171494    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-dzcvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Nov 03 18:19:08.171807 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:08.171563    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dzcvs" podUID="7e4a915b-dd46-4d77-b425-8c37440fdaad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Nov 03 18:19:08.180265 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:08.180227    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-7jkn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Nov 03 18:19:08.180787 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:08.180287    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7jkn9" podUID="0e37d948-2760-4441-8d44-722000f7e5b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Nov 03 18:19:08.189654 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:08.189599    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-4jx7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Nov 03 18:19:08.189824 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:08.189672    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4jx7c" podUID="8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Nov 03 18:19:10.093529 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:10.093494    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-nd6sq"
Nov 03 18:19:10.095492 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:10.095467    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-nd6sq"
Nov 03 18:19:10.096634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:10.096595    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-nd6sq"
Nov 03 18:19:10.133728 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:10.133700    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7jkn9"
Nov 03 18:19:10.134563 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:10.134534    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7jkn9"
Nov 03 18:19:10.135380 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:10.135360    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7jkn9"
Nov 03 18:19:10.174876 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:10.174838    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-dzcvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:19:10.175256 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:10.175115    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dzcvs" podUID="7e4a915b-dd46-4d77-b425-8c37440fdaad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:19:10.176531 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:10.176504    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dzcvs"
Nov 03 18:19:10.176873 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:10.176847    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dzcvs"
Nov 03 18:19:10.177506 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:10.177487    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dzcvs"
Nov 03 18:19:10.189870 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:10.189837    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-4jx7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:19:10.190083 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:10.189891    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4jx7c" podUID="8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:19:10.202294 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:10.202261    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4jx7c"
Nov 03 18:19:10.202695 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:10.202664    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4jx7c"
Nov 03 18:19:10.203816 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:10.203794    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4jx7c"
Nov 03 18:19:15.878440 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:15.878353    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2256a988-kv7fn"]
Nov 03 18:19:15.883952 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:15.883926    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2256a988-kv7fn"
Nov 03 18:19:15.932517 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:15.932479    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:19:15.958134 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:15.958097    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds9h2\" (UniqueName: \"kubernetes.io/projected/06c7f90e-b35e-4c42-bc19-a98f8cd418fa-kube-api-access-ds9h2\") pod \"maas-api-revoke-sub-2256a988-kv7fn\" (UID: \"06c7f90e-b35e-4c42-bc19-a98f8cd418fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2256a988-kv7fn"
Nov 03 18:19:15.958318 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:15.958222    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06c7f90e-b35e-4c42-bc19-a98f8cd418fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2256a988-kv7fn\" (UID: \"06c7f90e-b35e-4c42-bc19-a98f8cd418fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2256a988-kv7fn"
Nov 03 18:19:15.958318 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:15.958263    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06c7f90e-b35e-4c42-bc19-a98f8cd418fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2256a988-kv7fn\" (UID: \"06c7f90e-b35e-4c42-bc19-a98f8cd418fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2256a988-kv7fn"
Nov 03 18:19:15.978376 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:15.978335    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:19:15.978549 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:15.978380    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:19:15.978549 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:15.978396    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:19:15.980215 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:15.980195    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2256a988-kv7fn"]
Nov 03 18:19:16.059335 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:16.059300    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06c7f90e-b35e-4c42-bc19-a98f8cd418fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2256a988-kv7fn\" (UID: \"06c7f90e-b35e-4c42-bc19-a98f8cd418fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2256a988-kv7fn"
Nov 03 18:19:16.059335 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:16.059343    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06c7f90e-b35e-4c42-bc19-a98f8cd418fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2256a988-kv7fn\" (UID: \"06c7f90e-b35e-4c42-bc19-a98f8cd418fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2256a988-kv7fn"
Nov 03 18:19:16.059574 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:16.059402    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds9h2\" (UniqueName: \"kubernetes.io/projected/06c7f90e-b35e-4c42-bc19-a98f8cd418fa-kube-api-access-ds9h2\") pod \"maas-api-revoke-sub-2256a988-kv7fn\" (UID: \"06c7f90e-b35e-4c42-bc19-a98f8cd418fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2256a988-kv7fn"
Nov 03 18:19:16.060121 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:16.060096    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06c7f90e-b35e-4c42-bc19-a98f8cd418fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2256a988-kv7fn\" (UID: \"06c7f90e-b35e-4c42-bc19-a98f8cd418fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2256a988-kv7fn"
Nov 03 18:19:16.061778 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:16.061757    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06c7f90e-b35e-4c42-bc19-a98f8cd418fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2256a988-kv7fn\" (UID: \"06c7f90e-b35e-4c42-bc19-a98f8cd418fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2256a988-kv7fn"
Nov 03 18:19:16.117198 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:16.117161    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds9h2\" (UniqueName: \"kubernetes.io/projected/06c7f90e-b35e-4c42-bc19-a98f8cd418fa-kube-api-access-ds9h2\") pod \"maas-api-revoke-sub-2256a988-kv7fn\" (UID: \"06c7f90e-b35e-4c42-bc19-a98f8cd418fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2256a988-kv7fn"
Nov 03 18:19:16.197920 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:16.197820    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2256a988-kv7fn"
Nov 03 18:19:16.567109 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:16.567072    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2256a988-kv7fn"]
Nov 03 18:19:16.573509 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:19:16.573479    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06c7f90e_b35e_4c42_bc19_a98f8cd418fa.slice/crio-8d770d6812a54e773bc0ff1b1a23033494a04a5349c31bfc2e90e594f6de4ab5 WatchSource:0}: Error finding container 8d770d6812a54e773bc0ff1b1a23033494a04a5349c31bfc2e90e594f6de4ab5: Status 404 returned error can't find the container with id 8d770d6812a54e773bc0ff1b1a23033494a04a5349c31bfc2e90e594f6de4ab5
Nov 03 18:19:16.812780 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:16.812741    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2256a988-kv7fn" event={"ID":"06c7f90e-b35e-4c42-bc19-a98f8cd418fa","Type":"ContainerStarted","Data":"8d770d6812a54e773bc0ff1b1a23033494a04a5349c31bfc2e90e594f6de4ab5"}
Nov 03 18:19:17.819589 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:17.819556    5315 generic.go:356] "Generic (PLEG): container finished" podID="06c7f90e-b35e-4c42-bc19-a98f8cd418fa" containerID="a4f2216c44649b4a11d31532fab0679df82f7bfde1ccf513b3f8b140e2c9e19d" exitCode=0
Nov 03 18:19:17.819987 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:17.819649    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2256a988-kv7fn" event={"ID":"06c7f90e-b35e-4c42-bc19-a98f8cd418fa","Type":"ContainerDied","Data":"a4f2216c44649b4a11d31532fab0679df82f7bfde1ccf513b3f8b140e2c9e19d"}
Nov 03 18:19:18.963208 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:18.963182    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2256a988-kv7fn"
Nov 03 18:19:19.089704 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:19.089598    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06c7f90e-b35e-4c42-bc19-a98f8cd418fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06c7f90e-b35e-4c42-bc19-a98f8cd418fa-maas-api-xks-ca\") pod \"06c7f90e-b35e-4c42-bc19-a98f8cd418fa\" (UID: \"06c7f90e-b35e-4c42-bc19-a98f8cd418fa\") "
Nov 03 18:19:19.089704 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:19.089690    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06c7f90e-b35e-4c42-bc19-a98f8cd418fa-kube-api-access-ds9h2\" (UniqueName: \"kubernetes.io/projected/06c7f90e-b35e-4c42-bc19-a98f8cd418fa-kube-api-access-ds9h2\") pod \"06c7f90e-b35e-4c42-bc19-a98f8cd418fa\" (UID: \"06c7f90e-b35e-4c42-bc19-a98f8cd418fa\") "
Nov 03 18:19:19.089962 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:19.089790    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06c7f90e-b35e-4c42-bc19-a98f8cd418fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06c7f90e-b35e-4c42-bc19-a98f8cd418fa-maas-api-openshift-ca\") pod \"06c7f90e-b35e-4c42-bc19-a98f8cd418fa\" (UID: \"06c7f90e-b35e-4c42-bc19-a98f8cd418fa\") "
Nov 03 18:19:19.090178 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:19.090152    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06c7f90e-b35e-4c42-bc19-a98f8cd418fa-maas-api-openshift-ca" pod "06c7f90e-b35e-4c42-bc19-a98f8cd418fa" (UID: "06c7f90e-b35e-4c42-bc19-a98f8cd418fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:19:19.091767 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:19.091739    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06c7f90e-b35e-4c42-bc19-a98f8cd418fa-maas-api-xks-ca" pod "06c7f90e-b35e-4c42-bc19-a98f8cd418fa" (UID: "06c7f90e-b35e-4c42-bc19-a98f8cd418fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:19:19.091912 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:19.091892    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06c7f90e-b35e-4c42-bc19-a98f8cd418fa-kube-api-access-ds9h2" pod "06c7f90e-b35e-4c42-bc19-a98f8cd418fa" (UID: "06c7f90e-b35e-4c42-bc19-a98f8cd418fa"). InnerVolumeSpecName "kube-api-access-ds9h2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:19:19.190799 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:19.190757    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06c7f90e-b35e-4c42-bc19-a98f8cd418fa-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:19:19.190799 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:19.190796    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06c7f90e-b35e-4c42-bc19-a98f8cd418fa-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:19:19.190799 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:19.190809    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds9h2\" (UniqueName: \"kubernetes.io/projected/06c7f90e-b35e-4c42-bc19-a98f8cd418fa-kube-api-access-ds9h2\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:19:19.831374 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:19.831339    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2256a988-kv7fn"
Nov 03 18:19:19.831574 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:19.831340    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2256a988-kv7fn" event={"ID":"06c7f90e-b35e-4c42-bc19-a98f8cd418fa","Type":"ContainerDied","Data":"8d770d6812a54e773bc0ff1b1a23033494a04a5349c31bfc2e90e594f6de4ab5"}
Nov 03 18:19:19.831574 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:19.831476    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d770d6812a54e773bc0ff1b1a23033494a04a5349c31bfc2e90e594f6de4ab5"
Nov 03 18:19:24.576591 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:19:24.576552    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:19:27.132970 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:27.132932    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ef9633-22gsl"]
Nov 03 18:19:27.133528 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:27.133505    5315 cpu_manager.go:414] "containerMap: removing container" podUID="06c7f90e-b35e-4c42-bc19-a98f8cd418fa" containerName="revoke-keys"
Nov 03 18:19:27.133631 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:27.133532    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06c7f90e-b35e-4c42-bc19-a98f8cd418fa" containerName="revoke-keys"
Nov 03 18:19:27.133703 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:27.133660    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="06c7f90e-b35e-4c42-bc19-a98f8cd418fa" containerName="revoke-keys"
Nov 03 18:19:27.136682 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:27.136660    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ef9633-22gsl"
Nov 03 18:19:27.231187 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:27.231149    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:19:27.231187 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:27.231177    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:19:27.231485 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:27.231232    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:19:27.231485 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:27.231256    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:19:27.233172 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:27.233146    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ef9633-22gsl"]
Nov 03 18:19:27.264651 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:27.264587    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2pms\" (UniqueName: \"kubernetes.io/projected/4a920b51-bd19-4077-bee3-b7cf7d848da7-kube-api-access-p2pms\") pod \"maas-api-revoke-keys-e2e-worker-w1-ef9633-22gsl\" (UID: \"4a920b51-bd19-4077-bee3-b7cf7d848da7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ef9633-22gsl"
Nov 03 18:19:27.264841 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:27.264769    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a920b51-bd19-4077-bee3-b7cf7d848da7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-ef9633-22gsl\" (UID: \"4a920b51-bd19-4077-bee3-b7cf7d848da7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ef9633-22gsl"
Nov 03 18:19:27.264841 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:27.264809    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a920b51-bd19-4077-bee3-b7cf7d848da7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-ef9633-22gsl\" (UID: \"4a920b51-bd19-4077-bee3-b7cf7d848da7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ef9633-22gsl"
Nov 03 18:19:27.365559 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:27.365511    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a920b51-bd19-4077-bee3-b7cf7d848da7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-ef9633-22gsl\" (UID: \"4a920b51-bd19-4077-bee3-b7cf7d848da7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ef9633-22gsl"
Nov 03 18:19:27.365806 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:27.365568    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a920b51-bd19-4077-bee3-b7cf7d848da7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-ef9633-22gsl\" (UID: \"4a920b51-bd19-4077-bee3-b7cf7d848da7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ef9633-22gsl"
Nov 03 18:19:27.365806 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:27.365662    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2pms\" (UniqueName: \"kubernetes.io/projected/4a920b51-bd19-4077-bee3-b7cf7d848da7-kube-api-access-p2pms\") pod \"maas-api-revoke-keys-e2e-worker-w1-ef9633-22gsl\" (UID: \"4a920b51-bd19-4077-bee3-b7cf7d848da7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ef9633-22gsl"
Nov 03 18:19:27.366233 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:27.366206    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a920b51-bd19-4077-bee3-b7cf7d848da7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-ef9633-22gsl\" (UID: \"4a920b51-bd19-4077-bee3-b7cf7d848da7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ef9633-22gsl"
Nov 03 18:19:27.368148 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:27.368126    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a920b51-bd19-4077-bee3-b7cf7d848da7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-ef9633-22gsl\" (UID: \"4a920b51-bd19-4077-bee3-b7cf7d848da7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ef9633-22gsl"
Nov 03 18:19:27.423110 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:27.423034    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2pms\" (UniqueName: \"kubernetes.io/projected/4a920b51-bd19-4077-bee3-b7cf7d848da7-kube-api-access-p2pms\") pod \"maas-api-revoke-keys-e2e-worker-w1-ef9633-22gsl\" (UID: \"4a920b51-bd19-4077-bee3-b7cf7d848da7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ef9633-22gsl"
Nov 03 18:19:27.449724 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:27.449686    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ef9633-22gsl"
Nov 03 18:19:27.820753 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:27.820717    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ef9633-22gsl"]
Nov 03 18:19:27.827110 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:19:27.827082    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a920b51_bd19_4077_bee3_b7cf7d848da7.slice/crio-150c7f3be7bb5924f626fa649c960cf091f737645963c3ce08300764a26e1bdb WatchSource:0}: Error finding container 150c7f3be7bb5924f626fa649c960cf091f737645963c3ce08300764a26e1bdb: Status 404 returned error can't find the container with id 150c7f3be7bb5924f626fa649c960cf091f737645963c3ce08300764a26e1bdb
Nov 03 18:19:27.876191 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:27.876150    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ef9633-22gsl" event={"ID":"4a920b51-bd19-4077-bee3-b7cf7d848da7","Type":"ContainerStarted","Data":"150c7f3be7bb5924f626fa649c960cf091f737645963c3ce08300764a26e1bdb"}
Nov 03 18:19:28.045504 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:28.045470    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dff4788-m9mbb"]
Nov 03 18:19:28.048129 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:28.048098    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dff4788-m9mbb"]
Nov 03 18:19:28.884823 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:28.884792    5315 generic.go:356] "Generic (PLEG): container finished" podID="4a920b51-bd19-4077-bee3-b7cf7d848da7" containerID="3d5c97b2587474fc715cf28ce9fe4efa2d96afe5c41619726466df69c752e137" exitCode=0
Nov 03 18:19:28.885258 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:28.884892    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ef9633-22gsl" event={"ID":"4a920b51-bd19-4077-bee3-b7cf7d848da7","Type":"ContainerDied","Data":"3d5c97b2587474fc715cf28ce9fe4efa2d96afe5c41619726466df69c752e137"}
Nov 03 18:19:29.046193 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:29.046162    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37b4fe53-tb2hn"]
Nov 03 18:19:29.049253 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:29.049218    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37b4fe53-tb2hn"]
Nov 03 18:19:29.102448 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:29.102412    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="959ef213-8356-4ab2-8649-0378e33730d5" path="/var/lib/kubelet/pods/959ef213-8356-4ab2-8649-0378e33730d5/volumes"
Nov 03 18:19:29.102894 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:29.102879    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af" path="/var/lib/kubelet/pods/b5e7afbc-2298-4fb8-bd6a-4c13b8cd95af/volumes"
Nov 03 18:19:30.091424 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:30.091387    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ef9633-22gsl"
Nov 03 18:19:30.193236 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:30.193118    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a920b51-bd19-4077-bee3-b7cf7d848da7-kube-api-access-p2pms\" (UniqueName: \"kubernetes.io/projected/4a920b51-bd19-4077-bee3-b7cf7d848da7-kube-api-access-p2pms\") pod \"4a920b51-bd19-4077-bee3-b7cf7d848da7\" (UID: \"4a920b51-bd19-4077-bee3-b7cf7d848da7\") "
Nov 03 18:19:30.193434 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:30.193243    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a920b51-bd19-4077-bee3-b7cf7d848da7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a920b51-bd19-4077-bee3-b7cf7d848da7-maas-api-xks-ca\") pod \"4a920b51-bd19-4077-bee3-b7cf7d848da7\" (UID: \"4a920b51-bd19-4077-bee3-b7cf7d848da7\") "
Nov 03 18:19:30.193434 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:30.193309    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a920b51-bd19-4077-bee3-b7cf7d848da7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a920b51-bd19-4077-bee3-b7cf7d848da7-maas-api-openshift-ca\") pod \"4a920b51-bd19-4077-bee3-b7cf7d848da7\" (UID: \"4a920b51-bd19-4077-bee3-b7cf7d848da7\") "
Nov 03 18:19:30.193808 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:30.193766    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a920b51-bd19-4077-bee3-b7cf7d848da7-maas-api-openshift-ca" pod "4a920b51-bd19-4077-bee3-b7cf7d848da7" (UID: "4a920b51-bd19-4077-bee3-b7cf7d848da7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:19:30.196498 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:30.196461    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a920b51-bd19-4077-bee3-b7cf7d848da7-kube-api-access-p2pms" pod "4a920b51-bd19-4077-bee3-b7cf7d848da7" (UID: "4a920b51-bd19-4077-bee3-b7cf7d848da7"). InnerVolumeSpecName "kube-api-access-p2pms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:19:30.196652 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:30.196593    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a920b51-bd19-4077-bee3-b7cf7d848da7-maas-api-xks-ca" pod "4a920b51-bd19-4077-bee3-b7cf7d848da7" (UID: "4a920b51-bd19-4077-bee3-b7cf7d848da7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:19:30.295218 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:30.295160    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2pms\" (UniqueName: \"kubernetes.io/projected/4a920b51-bd19-4077-bee3-b7cf7d848da7-kube-api-access-p2pms\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:19:30.295218 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:30.295215    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a920b51-bd19-4077-bee3-b7cf7d848da7-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:19:30.295521 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:30.295234    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a920b51-bd19-4077-bee3-b7cf7d848da7-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:19:30.919219 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:30.919179    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ef9633-22gsl" event={"ID":"4a920b51-bd19-4077-bee3-b7cf7d848da7","Type":"ContainerDied","Data":"150c7f3be7bb5924f626fa649c960cf091f737645963c3ce08300764a26e1bdb"}
Nov 03 18:19:30.919601 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:30.919580    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="150c7f3be7bb5924f626fa649c960cf091f737645963c3ce08300764a26e1bdb"
Nov 03 18:19:30.919807 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:30.919794    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ef9633-22gsl"
Nov 03 18:19:38.635173 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:38.635133    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-ef9633-769c9f5f8f-mj4br"]
Nov 03 18:19:41.759218 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:41.759180    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa531960-p6vw5"]
Nov 03 18:19:41.759643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:41.759573    5315 cpu_manager.go:414] "containerMap: removing container" podUID="4a920b51-bd19-4077-bee3-b7cf7d848da7" containerName="revoke-keys"
Nov 03 18:19:41.759643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:41.759584    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a920b51-bd19-4077-bee3-b7cf7d848da7" containerName="revoke-keys"
Nov 03 18:19:41.759732 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:41.759660    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a920b51-bd19-4077-bee3-b7cf7d848da7" containerName="revoke-keys"
Nov 03 18:19:41.764160 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:41.764130    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa531960-p6vw5"
Nov 03 18:19:41.836801 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:41.836766    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:19:41.842227 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:41.842192    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prxnz\" (UniqueName: \"kubernetes.io/projected/9f7a62e2-84ad-4c09-980e-d69d77d69531-kube-api-access-prxnz\") pod \"maas-api-revoke-sub-aa531960-p6vw5\" (UID: \"9f7a62e2-84ad-4c09-980e-d69d77d69531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa531960-p6vw5"
Nov 03 18:19:41.842390 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:41.842278    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f7a62e2-84ad-4c09-980e-d69d77d69531-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa531960-p6vw5\" (UID: \"9f7a62e2-84ad-4c09-980e-d69d77d69531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa531960-p6vw5"
Nov 03 18:19:41.842390 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:41.842329    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f7a62e2-84ad-4c09-980e-d69d77d69531-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa531960-p6vw5\" (UID: \"9f7a62e2-84ad-4c09-980e-d69d77d69531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa531960-p6vw5"
Nov 03 18:19:41.883133 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:41.883097    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:19:41.883133 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:41.883111    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:19:41.883133 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:41.883114    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:19:41.885123 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:41.885099    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa531960-p6vw5"]
Nov 03 18:19:41.943895 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:41.943853    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f7a62e2-84ad-4c09-980e-d69d77d69531-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa531960-p6vw5\" (UID: \"9f7a62e2-84ad-4c09-980e-d69d77d69531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa531960-p6vw5"
Nov 03 18:19:41.944118 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:41.943976    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prxnz\" (UniqueName: \"kubernetes.io/projected/9f7a62e2-84ad-4c09-980e-d69d77d69531-kube-api-access-prxnz\") pod \"maas-api-revoke-sub-aa531960-p6vw5\" (UID: \"9f7a62e2-84ad-4c09-980e-d69d77d69531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa531960-p6vw5"
Nov 03 18:19:41.944118 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:41.944000    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f7a62e2-84ad-4c09-980e-d69d77d69531-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa531960-p6vw5\" (UID: \"9f7a62e2-84ad-4c09-980e-d69d77d69531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa531960-p6vw5"
Nov 03 18:19:41.944669 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:41.944645    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f7a62e2-84ad-4c09-980e-d69d77d69531-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa531960-p6vw5\" (UID: \"9f7a62e2-84ad-4c09-980e-d69d77d69531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa531960-p6vw5"
Nov 03 18:19:41.946638 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:41.946594    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f7a62e2-84ad-4c09-980e-d69d77d69531-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa531960-p6vw5\" (UID: \"9f7a62e2-84ad-4c09-980e-d69d77d69531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa531960-p6vw5"
Nov 03 18:19:42.003655 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:42.003601    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prxnz\" (UniqueName: \"kubernetes.io/projected/9f7a62e2-84ad-4c09-980e-d69d77d69531-kube-api-access-prxnz\") pod \"maas-api-revoke-sub-aa531960-p6vw5\" (UID: \"9f7a62e2-84ad-4c09-980e-d69d77d69531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa531960-p6vw5"
Nov 03 18:19:42.077408 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:42.077359    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa531960-p6vw5"
Nov 03 18:19:42.448895 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:42.448796    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa531960-p6vw5"]
Nov 03 18:19:42.455539 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:19:42.455500    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f7a62e2_84ad_4c09_980e_d69d77d69531.slice/crio-7d761b399d0bdd604dbf4c9dde9a91cceeafa234c394d8b3d67c80026f4ccc84 WatchSource:0}: Error finding container 7d761b399d0bdd604dbf4c9dde9a91cceeafa234c394d8b3d67c80026f4ccc84: Status 404 returned error can't find the container with id 7d761b399d0bdd604dbf4c9dde9a91cceeafa234c394d8b3d67c80026f4ccc84
Nov 03 18:19:43.013907 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:43.013810    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa531960-p6vw5" event={"ID":"9f7a62e2-84ad-4c09-980e-d69d77d69531","Type":"ContainerStarted","Data":"da9f37a8193cc773b4cae8f2459baa369721dedb302a26f5b52087473f086ca1"}
Nov 03 18:19:43.013907 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:43.013876    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa531960-p6vw5" event={"ID":"9f7a62e2-84ad-4c09-980e-d69d77d69531","Type":"ContainerStarted","Data":"7d761b399d0bdd604dbf4c9dde9a91cceeafa234c394d8b3d67c80026f4ccc84"}
Nov 03 18:19:43.123125 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:43.123068    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa531960-p6vw5" podStartSLOduration=2.123052619 podStartE2EDuration="2.123052619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:19:43.121536489 +0000 UTC m=+2022.137734557" watchObservedRunningTime="2026-11-03 18:19:43.123052619 +0000 UTC m=+2022.139250687"
Nov 03 18:19:43.636201 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:43.636079    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-ef9633-769c9f5f8f-mj4br" podUID="e8f65f51-9bb6-4242-a9d6-9a2934f06f84" containerName="payload-processing" containerID="cri-o://d03879416fa4f2f391ba7f5c5c3f6b85442a7cd46b8b6327ba8f76d6917947cb" gracePeriod=25
Nov 03 18:19:44.020702 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:44.020592    5315 generic.go:356] "Generic (PLEG): container finished" podID="9f7a62e2-84ad-4c09-980e-d69d77d69531" containerID="da9f37a8193cc773b4cae8f2459baa369721dedb302a26f5b52087473f086ca1" exitCode=0
Nov 03 18:19:44.020702 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:44.020642    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa531960-p6vw5" event={"ID":"9f7a62e2-84ad-4c09-980e-d69d77d69531","Type":"ContainerDied","Data":"da9f37a8193cc773b4cae8f2459baa369721dedb302a26f5b52087473f086ca1"}
Nov 03 18:19:44.452427 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:44.452389    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-ef9633-769c9f5f8f-mj4br" podUID="e8f65f51-9bb6-4242-a9d6-9a2934f06f84" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:19:45.166567 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:45.166537    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa531960-p6vw5"
Nov 03 18:19:45.275670 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:45.275627    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f7a62e2-84ad-4c09-980e-d69d77d69531-kube-api-access-prxnz\" (UniqueName: \"kubernetes.io/projected/9f7a62e2-84ad-4c09-980e-d69d77d69531-kube-api-access-prxnz\") pod \"9f7a62e2-84ad-4c09-980e-d69d77d69531\" (UID: \"9f7a62e2-84ad-4c09-980e-d69d77d69531\") "
Nov 03 18:19:45.275857 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:45.275750    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f7a62e2-84ad-4c09-980e-d69d77d69531-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f7a62e2-84ad-4c09-980e-d69d77d69531-maas-api-xks-ca\") pod \"9f7a62e2-84ad-4c09-980e-d69d77d69531\" (UID: \"9f7a62e2-84ad-4c09-980e-d69d77d69531\") "
Nov 03 18:19:45.275857 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:45.275786    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f7a62e2-84ad-4c09-980e-d69d77d69531-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f7a62e2-84ad-4c09-980e-d69d77d69531-maas-api-openshift-ca\") pod \"9f7a62e2-84ad-4c09-980e-d69d77d69531\" (UID: \"9f7a62e2-84ad-4c09-980e-d69d77d69531\") "
Nov 03 18:19:45.276191 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:45.276165    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f7a62e2-84ad-4c09-980e-d69d77d69531-maas-api-openshift-ca" pod "9f7a62e2-84ad-4c09-980e-d69d77d69531" (UID: "9f7a62e2-84ad-4c09-980e-d69d77d69531"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:19:45.277852 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:45.277789    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f7a62e2-84ad-4c09-980e-d69d77d69531-kube-api-access-prxnz" pod "9f7a62e2-84ad-4c09-980e-d69d77d69531" (UID: "9f7a62e2-84ad-4c09-980e-d69d77d69531"). InnerVolumeSpecName "kube-api-access-prxnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:19:45.277852 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:45.277827    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f7a62e2-84ad-4c09-980e-d69d77d69531-maas-api-xks-ca" pod "9f7a62e2-84ad-4c09-980e-d69d77d69531" (UID: "9f7a62e2-84ad-4c09-980e-d69d77d69531"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:19:45.376729 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:45.376691    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prxnz\" (UniqueName: \"kubernetes.io/projected/9f7a62e2-84ad-4c09-980e-d69d77d69531-kube-api-access-prxnz\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:19:45.376729 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:45.376724    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f7a62e2-84ad-4c09-980e-d69d77d69531-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:19:45.376729 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:45.376734    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f7a62e2-84ad-4c09-980e-d69d77d69531-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:19:46.032706 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:46.032668    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa531960-p6vw5"
Nov 03 18:19:46.032706 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:46.032674    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa531960-p6vw5" event={"ID":"9f7a62e2-84ad-4c09-980e-d69d77d69531","Type":"ContainerDied","Data":"7d761b399d0bdd604dbf4c9dde9a91cceeafa234c394d8b3d67c80026f4ccc84"}
Nov 03 18:19:46.032928 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:46.032730    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d761b399d0bdd604dbf4c9dde9a91cceeafa234c394d8b3d67c80026f4ccc84"
Nov 03 18:19:54.452852 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:54.452814    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-ef9633-769c9f5f8f-mj4br" podUID="e8f65f51-9bb6-4242-a9d6-9a2934f06f84" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:19:54.577361 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:19:54.577320    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:19:59.242689 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:59.242650    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ef9633-22gsl"]
Nov 03 18:19:59.288836 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:19:59.288797    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ef9633-22gsl"]
Nov 03 18:20:01.102703 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:01.102660    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a920b51-bd19-4077-bee3-b7cf7d848da7" path="/var/lib/kubelet/pods/4a920b51-bd19-4077-bee3-b7cf7d848da7/volumes"
Nov 03 18:20:03.886014 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:03.885986    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-ef9633-769c9f5f8f-mj4br"
Nov 03 18:20:03.950024 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:03.949917    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8f65f51-9bb6-4242-a9d6-9a2934f06f84-kube-api-access-mwh7k\" (UniqueName: \"kubernetes.io/projected/e8f65f51-9bb6-4242-a9d6-9a2934f06f84-kube-api-access-mwh7k\") pod \"e8f65f51-9bb6-4242-a9d6-9a2934f06f84\" (UID: \"e8f65f51-9bb6-4242-a9d6-9a2934f06f84\") "
Nov 03 18:20:03.950223 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:03.950050    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8f65f51-9bb6-4242-a9d6-9a2934f06f84-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8f65f51-9bb6-4242-a9d6-9a2934f06f84-plugins-config-volume\") pod \"e8f65f51-9bb6-4242-a9d6-9a2934f06f84\" (UID: \"e8f65f51-9bb6-4242-a9d6-9a2934f06f84\") "
Nov 03 18:20:03.950423 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:03.950398    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8f65f51-9bb6-4242-a9d6-9a2934f06f84-plugins-config-volume" pod "e8f65f51-9bb6-4242-a9d6-9a2934f06f84" (UID: "e8f65f51-9bb6-4242-a9d6-9a2934f06f84"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:20:03.952190 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:03.952155    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8f65f51-9bb6-4242-a9d6-9a2934f06f84-kube-api-access-mwh7k" pod "e8f65f51-9bb6-4242-a9d6-9a2934f06f84" (UID: "e8f65f51-9bb6-4242-a9d6-9a2934f06f84"). InnerVolumeSpecName "kube-api-access-mwh7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:20:04.050805 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:04.050761    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwh7k\" (UniqueName: \"kubernetes.io/projected/e8f65f51-9bb6-4242-a9d6-9a2934f06f84-kube-api-access-mwh7k\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:04.050805 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:04.050800    5315 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8f65f51-9bb6-4242-a9d6-9a2934f06f84-plugins-config-volume\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:04.140331 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:04.140199    5315 generic.go:356] "Generic (PLEG): container finished" podID="e8f65f51-9bb6-4242-a9d6-9a2934f06f84" containerID="d03879416fa4f2f391ba7f5c5c3f6b85442a7cd46b8b6327ba8f76d6917947cb" exitCode=0
Nov 03 18:20:04.140331 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:04.140277    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-ef9633-769c9f5f8f-mj4br"
Nov 03 18:20:04.140625 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:04.140269    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-ef9633-769c9f5f8f-mj4br" event={"ID":"e8f65f51-9bb6-4242-a9d6-9a2934f06f84","Type":"ContainerDied","Data":"d03879416fa4f2f391ba7f5c5c3f6b85442a7cd46b8b6327ba8f76d6917947cb"}
Nov 03 18:20:04.140625 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:04.140426    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-ef9633-769c9f5f8f-mj4br" event={"ID":"e8f65f51-9bb6-4242-a9d6-9a2934f06f84","Type":"ContainerDied","Data":"fe6c235ce2f3ffc9f03b6b506b5f8b7800edae5b4aafddb4f27350dc2932a1ef"}
Nov 03 18:20:04.140625 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:04.140460    5315 scope.go:122] "RemoveContainer" containerID="d03879416fa4f2f391ba7f5c5c3f6b85442a7cd46b8b6327ba8f76d6917947cb"
Nov 03 18:20:04.150454 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:04.150432    5315 scope.go:122] "RemoveContainer" containerID="d03879416fa4f2f391ba7f5c5c3f6b85442a7cd46b8b6327ba8f76d6917947cb"
Nov 03 18:20:04.150810 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:20:04.150789    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d03879416fa4f2f391ba7f5c5c3f6b85442a7cd46b8b6327ba8f76d6917947cb\": container with ID starting with d03879416fa4f2f391ba7f5c5c3f6b85442a7cd46b8b6327ba8f76d6917947cb not found: ID does not exist" containerID="d03879416fa4f2f391ba7f5c5c3f6b85442a7cd46b8b6327ba8f76d6917947cb"
Nov 03 18:20:04.150888 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:04.150819    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d03879416fa4f2f391ba7f5c5c3f6b85442a7cd46b8b6327ba8f76d6917947cb"} err="failed to get container status \"d03879416fa4f2f391ba7f5c5c3f6b85442a7cd46b8b6327ba8f76d6917947cb\": rpc error: code = NotFound desc = could not find container \"d03879416fa4f2f391ba7f5c5c3f6b85442a7cd46b8b6327ba8f76d6917947cb\": container with ID starting with d03879416fa4f2f391ba7f5c5c3f6b85442a7cd46b8b6327ba8f76d6917947cb not found: ID does not exist"
Nov 03 18:20:04.293261 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:04.293224    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-ef9633-769c9f5f8f-mj4br"]
Nov 03 18:20:04.309785 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:04.309749    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-ef9633-769c9f5f8f-mj4br"]
Nov 03 18:20:05.104087 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:05.104047    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8f65f51-9bb6-4242-a9d6-9a2934f06f84" path="/var/lib/kubelet/pods/e8f65f51-9bb6-4242-a9d6-9a2934f06f84/volumes"
Nov 03 18:20:05.271277 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:05.271244    5315 scope.go:122] "RemoveContainer" containerID="d9cebf1e4e089f884819ad7303ec8ac0333f947756864677b1cfc1c47eebd260"
Nov 03 18:20:05.280064 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:05.280041    5315 scope.go:122] "RemoveContainer" containerID="cfd88ada69ad0317fe2bd1b799dc75b9d201d139841d4cb4dfc5bf06702e4c16"
Nov 03 18:20:05.289063 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:05.289040    5315 scope.go:122] "RemoveContainer" containerID="c4ba43618a3b724d6ee5861a3cdb4ac8c748f9a2f19b4ef34518c01273caa558"
Nov 03 18:20:05.297397 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:05.297377    5315 scope.go:122] "RemoveContainer" containerID="0ef799abf00fafd0cd7246531d1810eb64a05b0e118cc960bc857b03e9059a44"
Nov 03 18:20:05.305759 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:05.305737    5315 scope.go:122] "RemoveContainer" containerID="93aec41d7cb9abdbae904ffea4b5cf4e6b61228c984e0ad21ef4fb53513b4677"
Nov 03 18:20:05.314538 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:05.314515    5315 scope.go:122] "RemoveContainer" containerID="03fca753ee86d38e5a9d763b20ba7e1e9240404937517e3ce0d54fc53c78b1f0"
Nov 03 18:20:05.322911 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:05.322884    5315 scope.go:122] "RemoveContainer" containerID="eed7385d3e244bb7b2aa09105e3dfd35ae02112d466bf21436ed51ce52a4edcc"
Nov 03 18:20:06.964322 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:06.964285    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855"]
Nov 03 18:20:06.964871 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:06.964552    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855" podUID="dd5c9fcf-e132-42e2-b031-c608cfd0b392" containerName="istio-proxy" containerID="cri-o://3808d426e0bdb3213501e57a43960bc2d0e77c720136cd88efdf64fcd9872873" gracePeriod=30
Nov 03 18:20:12.190094 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:12.190052    5315 generic.go:356] "Generic (PLEG): container finished" podID="dd5c9fcf-e132-42e2-b031-c608cfd0b392" containerID="3808d426e0bdb3213501e57a43960bc2d0e77c720136cd88efdf64fcd9872873" exitCode=0
Nov 03 18:20:12.190537 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:12.190117    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855" event={"ID":"dd5c9fcf-e132-42e2-b031-c608cfd0b392","Type":"ContainerDied","Data":"3808d426e0bdb3213501e57a43960bc2d0e77c720136cd88efdf64fcd9872873"}
Nov 03 18:20:12.190537 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:12.190186    5315 scope.go:122] "RemoveContainer" containerID="f2a6ea1f91953839e03ec6653e0e9e5ef8b0e68ec3835cf1c567f56ff182207a"
Nov 03 18:20:12.225003 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:12.224971    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855"
Nov 03 18:20:12.342455 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:12.342409    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd5c9fcf-e132-42e2-b031-c608cfd0b392-istio-token\" (UniqueName: \"kubernetes.io/projected/dd5c9fcf-e132-42e2-b031-c608cfd0b392-istio-token\") pod \"dd5c9fcf-e132-42e2-b031-c608cfd0b392\" (UID: \"dd5c9fcf-e132-42e2-b031-c608cfd0b392\") "
Nov 03 18:20:12.342674 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:12.342489    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd5c9fcf-e132-42e2-b031-c608cfd0b392-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd5c9fcf-e132-42e2-b031-c608cfd0b392-credential-socket\") pod \"dd5c9fcf-e132-42e2-b031-c608cfd0b392\" (UID: \"dd5c9fcf-e132-42e2-b031-c608cfd0b392\") "
Nov 03 18:20:12.342674 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:12.342510    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd5c9fcf-e132-42e2-b031-c608cfd0b392-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd5c9fcf-e132-42e2-b031-c608cfd0b392-istiod-ca-cert\") pod \"dd5c9fcf-e132-42e2-b031-c608cfd0b392\" (UID: \"dd5c9fcf-e132-42e2-b031-c608cfd0b392\") "
Nov 03 18:20:12.342674 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:12.342528    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd5c9fcf-e132-42e2-b031-c608cfd0b392-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd5c9fcf-e132-42e2-b031-c608cfd0b392-istio-envoy\") pod \"dd5c9fcf-e132-42e2-b031-c608cfd0b392\" (UID: \"dd5c9fcf-e132-42e2-b031-c608cfd0b392\") "
Nov 03 18:20:12.342674 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:12.342585    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd5c9fcf-e132-42e2-b031-c608cfd0b392-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd5c9fcf-e132-42e2-b031-c608cfd0b392-workload-socket\") pod \"dd5c9fcf-e132-42e2-b031-c608cfd0b392\" (UID: \"dd5c9fcf-e132-42e2-b031-c608cfd0b392\") "
Nov 03 18:20:12.342674 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:12.342643    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd5c9fcf-e132-42e2-b031-c608cfd0b392-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd5c9fcf-e132-42e2-b031-c608cfd0b392-workload-certs\") pod \"dd5c9fcf-e132-42e2-b031-c608cfd0b392\" (UID: \"dd5c9fcf-e132-42e2-b031-c608cfd0b392\") "
Nov 03 18:20:12.342989 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:12.342702    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd5c9fcf-e132-42e2-b031-c608cfd0b392-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd5c9fcf-e132-42e2-b031-c608cfd0b392-istio-data\") pod \"dd5c9fcf-e132-42e2-b031-c608cfd0b392\" (UID: \"dd5c9fcf-e132-42e2-b031-c608cfd0b392\") "
Nov 03 18:20:12.342989 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:12.342743    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd5c9fcf-e132-42e2-b031-c608cfd0b392-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd5c9fcf-e132-42e2-b031-c608cfd0b392-istio-podinfo\") pod \"dd5c9fcf-e132-42e2-b031-c608cfd0b392\" (UID: \"dd5c9fcf-e132-42e2-b031-c608cfd0b392\") "
Nov 03 18:20:12.342989 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:12.342812    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd5c9fcf-e132-42e2-b031-c608cfd0b392-kube-api-access-2x6c4\" (UniqueName: \"kubernetes.io/projected/dd5c9fcf-e132-42e2-b031-c608cfd0b392-kube-api-access-2x6c4\") pod \"dd5c9fcf-e132-42e2-b031-c608cfd0b392\" (UID: \"dd5c9fcf-e132-42e2-b031-c608cfd0b392\") "
Nov 03 18:20:12.342989 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:12.342867    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd5c9fcf-e132-42e2-b031-c608cfd0b392-credential-socket" pod "dd5c9fcf-e132-42e2-b031-c608cfd0b392" (UID: "dd5c9fcf-e132-42e2-b031-c608cfd0b392"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:12.342989 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:12.342938    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd5c9fcf-e132-42e2-b031-c608cfd0b392-workload-socket" pod "dd5c9fcf-e132-42e2-b031-c608cfd0b392" (UID: "dd5c9fcf-e132-42e2-b031-c608cfd0b392"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:12.343278 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:12.343002    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd5c9fcf-e132-42e2-b031-c608cfd0b392-istiod-ca-cert" pod "dd5c9fcf-e132-42e2-b031-c608cfd0b392" (UID: "dd5c9fcf-e132-42e2-b031-c608cfd0b392"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:20:12.343278 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:12.343106    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd5c9fcf-e132-42e2-b031-c608cfd0b392-istio-data" pod "dd5c9fcf-e132-42e2-b031-c608cfd0b392" (UID: "dd5c9fcf-e132-42e2-b031-c608cfd0b392"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:12.343278 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:12.343221    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd5c9fcf-e132-42e2-b031-c608cfd0b392-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:12.343278 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:12.343268    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd5c9fcf-e132-42e2-b031-c608cfd0b392-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:12.343459 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:12.343286    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd5c9fcf-e132-42e2-b031-c608cfd0b392-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:12.343459 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:12.343301    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd5c9fcf-e132-42e2-b031-c608cfd0b392-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:12.343459 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:12.343317    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd5c9fcf-e132-42e2-b031-c608cfd0b392-workload-certs" pod "dd5c9fcf-e132-42e2-b031-c608cfd0b392" (UID: "dd5c9fcf-e132-42e2-b031-c608cfd0b392"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:12.345062 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:12.345030    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd5c9fcf-e132-42e2-b031-c608cfd0b392-istio-envoy" pod "dd5c9fcf-e132-42e2-b031-c608cfd0b392" (UID: "dd5c9fcf-e132-42e2-b031-c608cfd0b392"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:12.345184 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:12.345066    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd5c9fcf-e132-42e2-b031-c608cfd0b392-istio-token" pod "dd5c9fcf-e132-42e2-b031-c608cfd0b392" (UID: "dd5c9fcf-e132-42e2-b031-c608cfd0b392"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:20:12.345301 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:12.345280    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd5c9fcf-e132-42e2-b031-c608cfd0b392-istio-podinfo" pod "dd5c9fcf-e132-42e2-b031-c608cfd0b392" (UID: "dd5c9fcf-e132-42e2-b031-c608cfd0b392"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:20:12.345396 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:12.345318    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd5c9fcf-e132-42e2-b031-c608cfd0b392-kube-api-access-2x6c4" pod "dd5c9fcf-e132-42e2-b031-c608cfd0b392" (UID: "dd5c9fcf-e132-42e2-b031-c608cfd0b392"). InnerVolumeSpecName "kube-api-access-2x6c4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:20:12.444630 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:12.444579    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd5c9fcf-e132-42e2-b031-c608cfd0b392-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:12.444836 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:12.444642    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd5c9fcf-e132-42e2-b031-c608cfd0b392-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:12.444836 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:12.444661    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd5c9fcf-e132-42e2-b031-c608cfd0b392-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:12.444836 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:12.444673    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x6c4\" (UniqueName: \"kubernetes.io/projected/dd5c9fcf-e132-42e2-b031-c608cfd0b392-kube-api-access-2x6c4\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:12.444836 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:12.444684    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd5c9fcf-e132-42e2-b031-c608cfd0b392-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:13.196429 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:13.196392    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855" event={"ID":"dd5c9fcf-e132-42e2-b031-c608cfd0b392","Type":"ContainerDied","Data":"e3d6c6c99680f0561fed9780f40861c6e8802a42d3845b64572c85e476fcc88b"}
Nov 03 18:20:13.196429 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:13.196424    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855"
Nov 03 18:20:13.196945 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:13.196460    5315 scope.go:122] "RemoveContainer" containerID="3808d426e0bdb3213501e57a43960bc2d0e77c720136cd88efdf64fcd9872873"
Nov 03 18:20:13.396044 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:13.395953    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855"]
Nov 03 18:20:13.441902 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:13.441867    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-nb855"]
Nov 03 18:20:15.105996 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:15.105939    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd5c9fcf-e132-42e2-b031-c608cfd0b392" path="/var/lib/kubelet/pods/dd5c9fcf-e132-42e2-b031-c608cfd0b392/volumes"
Nov 03 18:20:17.061636 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:17.054430    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0278b261-8vrsm"]
Nov 03 18:20:17.120319 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:17.120285    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0278b261-8vrsm"]
Nov 03 18:20:18.044021 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:18.043983    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2853269b-rtts6"]
Nov 03 18:20:18.090004 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:18.089963    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2853269b-rtts6"]
Nov 03 18:20:19.103591 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:19.103559    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03b0d677-1c10-4757-8079-f00a09d4831e" path="/var/lib/kubelet/pods/03b0d677-1c10-4757-8079-f00a09d4831e/volumes"
Nov 03 18:20:19.104044 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:19.104029    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48396164-efa9-490e-9e12-13709493bd43" path="/var/lib/kubelet/pods/48396164-efa9-490e-9e12-13709493bd43/volumes"
Nov 03 18:20:24.577401 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:20:24.577361    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:20:37.044942 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:37.044900    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcf6b319-5txp6"]
Nov 03 18:20:37.091045 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:37.091008    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcf6b319-5txp6"]
Nov 03 18:20:37.102324 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:37.102290    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f66eb4b-1a62-4dad-b70a-8fdd8f715b70" path="/var/lib/kubelet/pods/0f66eb4b-1a62-4dad-b70a-8fdd8f715b70/volumes"
Nov 03 18:20:39.493741 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:39.493700    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-5fjng"]
Nov 03 18:20:39.494194 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:39.494076    5315 cpu_manager.go:414] "containerMap: removing container" podUID="dd5c9fcf-e132-42e2-b031-c608cfd0b392" containerName="istio-proxy"
Nov 03 18:20:39.494194 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:39.494089    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd5c9fcf-e132-42e2-b031-c608cfd0b392" containerName="istio-proxy"
Nov 03 18:20:39.494194 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:39.494108    5315 cpu_manager.go:414] "containerMap: removing container" podUID="e8f65f51-9bb6-4242-a9d6-9a2934f06f84" containerName="payload-processing"
Nov 03 18:20:39.494194 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:39.494114    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8f65f51-9bb6-4242-a9d6-9a2934f06f84" containerName="payload-processing"
Nov 03 18:20:39.494194 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:39.494125    5315 cpu_manager.go:414] "containerMap: removing container" podUID="9f7a62e2-84ad-4c09-980e-d69d77d69531" containerName="revoke-keys"
Nov 03 18:20:39.494194 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:39.494130    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f7a62e2-84ad-4c09-980e-d69d77d69531" containerName="revoke-keys"
Nov 03 18:20:39.494194 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:39.494192    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8f65f51-9bb6-4242-a9d6-9a2934f06f84" containerName="payload-processing"
Nov 03 18:20:39.494466 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:39.494202    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f7a62e2-84ad-4c09-980e-d69d77d69531" containerName="revoke-keys"
Nov 03 18:20:39.494466 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:39.494211    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd5c9fcf-e132-42e2-b031-c608cfd0b392" containerName="istio-proxy"
Nov 03 18:20:39.494466 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:39.494217    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd5c9fcf-e132-42e2-b031-c608cfd0b392" containerName="istio-proxy"
Nov 03 18:20:39.497451 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:39.497423    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-5fjng"
Nov 03 18:20:39.546213 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:39.546182    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:20:39.592134 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:39.592098    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:20:39.592134 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:39.592125    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:20:39.592367 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:39.592145    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:20:39.593594 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:39.593574    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-5fjng"]
Nov 03 18:20:39.637565 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:39.637523    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88de2dfb-9252-405d-b590-abf68ec6dbe9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-5fjng\" (UID: \"88de2dfb-9252-405d-b590-abf68ec6dbe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-5fjng"
Nov 03 18:20:39.637767 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:39.637574    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88de2dfb-9252-405d-b590-abf68ec6dbe9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-5fjng\" (UID: \"88de2dfb-9252-405d-b590-abf68ec6dbe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-5fjng"
Nov 03 18:20:39.637767 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:39.637690    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dn98\" (UniqueName: \"kubernetes.io/projected/88de2dfb-9252-405d-b590-abf68ec6dbe9-kube-api-access-4dn98\") pod \"maas-api-revoke-sub-85237e45-5fjng\" (UID: \"88de2dfb-9252-405d-b590-abf68ec6dbe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-5fjng"
Nov 03 18:20:39.738502 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:39.738465    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88de2dfb-9252-405d-b590-abf68ec6dbe9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-5fjng\" (UID: \"88de2dfb-9252-405d-b590-abf68ec6dbe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-5fjng"
Nov 03 18:20:39.738733 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:39.738515    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dn98\" (UniqueName: \"kubernetes.io/projected/88de2dfb-9252-405d-b590-abf68ec6dbe9-kube-api-access-4dn98\") pod \"maas-api-revoke-sub-85237e45-5fjng\" (UID: \"88de2dfb-9252-405d-b590-abf68ec6dbe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-5fjng"
Nov 03 18:20:39.738733 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:39.738655    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88de2dfb-9252-405d-b590-abf68ec6dbe9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-5fjng\" (UID: \"88de2dfb-9252-405d-b590-abf68ec6dbe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-5fjng"
Nov 03 18:20:39.739348 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:39.739327    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88de2dfb-9252-405d-b590-abf68ec6dbe9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-5fjng\" (UID: \"88de2dfb-9252-405d-b590-abf68ec6dbe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-5fjng"
Nov 03 18:20:39.740931 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:39.740910    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88de2dfb-9252-405d-b590-abf68ec6dbe9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-5fjng\" (UID: \"88de2dfb-9252-405d-b590-abf68ec6dbe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-5fjng"
Nov 03 18:20:39.795432 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:39.795400    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dn98\" (UniqueName: \"kubernetes.io/projected/88de2dfb-9252-405d-b590-abf68ec6dbe9-kube-api-access-4dn98\") pod \"maas-api-revoke-sub-85237e45-5fjng\" (UID: \"88de2dfb-9252-405d-b590-abf68ec6dbe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-5fjng"
Nov 03 18:20:39.810237 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:39.810211    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-5fjng"
Nov 03 18:20:40.178646 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:40.178551    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-5fjng"]
Nov 03 18:20:40.185373 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:20:40.185332    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88de2dfb_9252_405d_b590_abf68ec6dbe9.slice/crio-4a8ae3276073b0f6392a5c3509985779186ba1f0f39e6410dd823a22c4d09934 WatchSource:0}: Error finding container 4a8ae3276073b0f6392a5c3509985779186ba1f0f39e6410dd823a22c4d09934: Status 404 returned error can't find the container with id 4a8ae3276073b0f6392a5c3509985779186ba1f0f39e6410dd823a22c4d09934
Nov 03 18:20:40.358824 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:40.358788    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-5fjng" event={"ID":"88de2dfb-9252-405d-b590-abf68ec6dbe9","Type":"ContainerStarted","Data":"f817a122140fa1030af0b504f79bf1b433540d016ab69fd59ba88ffe1491b07b"}
Nov 03 18:20:40.358995 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:40.358851    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-5fjng" event={"ID":"88de2dfb-9252-405d-b590-abf68ec6dbe9","Type":"ContainerStarted","Data":"4a8ae3276073b0f6392a5c3509985779186ba1f0f39e6410dd823a22c4d09934"}
Nov 03 18:20:40.467436 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:40.467330    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-5fjng" podStartSLOduration=1.467316195 podStartE2EDuration="1.467316195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:20:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:20:40.466357285 +0000 UTC m=+2079.482555354" watchObservedRunningTime="2026-11-03 18:20:40.467316195 +0000 UTC m=+2079.483514263"
Nov 03 18:20:41.365976 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:41.365935    5315 generic.go:356] "Generic (PLEG): container finished" podID="88de2dfb-9252-405d-b590-abf68ec6dbe9" containerID="f817a122140fa1030af0b504f79bf1b433540d016ab69fd59ba88ffe1491b07b" exitCode=0
Nov 03 18:20:41.366386 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:41.366001    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-5fjng" event={"ID":"88de2dfb-9252-405d-b590-abf68ec6dbe9","Type":"ContainerDied","Data":"f817a122140fa1030af0b504f79bf1b433540d016ab69fd59ba88ffe1491b07b"}
Nov 03 18:20:42.501519 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:42.501493    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-5fjng"
Nov 03 18:20:42.566371 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:42.566331    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88de2dfb-9252-405d-b590-abf68ec6dbe9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88de2dfb-9252-405d-b590-abf68ec6dbe9-maas-api-xks-ca\") pod \"88de2dfb-9252-405d-b590-abf68ec6dbe9\" (UID: \"88de2dfb-9252-405d-b590-abf68ec6dbe9\") "
Nov 03 18:20:42.566555 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:42.566411    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88de2dfb-9252-405d-b590-abf68ec6dbe9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88de2dfb-9252-405d-b590-abf68ec6dbe9-maas-api-openshift-ca\") pod \"88de2dfb-9252-405d-b590-abf68ec6dbe9\" (UID: \"88de2dfb-9252-405d-b590-abf68ec6dbe9\") "
Nov 03 18:20:42.566555 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:42.566443    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88de2dfb-9252-405d-b590-abf68ec6dbe9-kube-api-access-4dn98\" (UniqueName: \"kubernetes.io/projected/88de2dfb-9252-405d-b590-abf68ec6dbe9-kube-api-access-4dn98\") pod \"88de2dfb-9252-405d-b590-abf68ec6dbe9\" (UID: \"88de2dfb-9252-405d-b590-abf68ec6dbe9\") "
Nov 03 18:20:42.566860 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:42.566828    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88de2dfb-9252-405d-b590-abf68ec6dbe9-maas-api-openshift-ca" pod "88de2dfb-9252-405d-b590-abf68ec6dbe9" (UID: "88de2dfb-9252-405d-b590-abf68ec6dbe9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:20:42.568454 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:42.568429    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88de2dfb-9252-405d-b590-abf68ec6dbe9-maas-api-xks-ca" pod "88de2dfb-9252-405d-b590-abf68ec6dbe9" (UID: "88de2dfb-9252-405d-b590-abf68ec6dbe9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:20:42.568545 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:42.568498    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88de2dfb-9252-405d-b590-abf68ec6dbe9-kube-api-access-4dn98" pod "88de2dfb-9252-405d-b590-abf68ec6dbe9" (UID: "88de2dfb-9252-405d-b590-abf68ec6dbe9"). InnerVolumeSpecName "kube-api-access-4dn98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:20:42.667379 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:42.667292    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88de2dfb-9252-405d-b590-abf68ec6dbe9-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:42.667379 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:42.667325    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88de2dfb-9252-405d-b590-abf68ec6dbe9-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:42.667379 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:42.667336    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dn98\" (UniqueName: \"kubernetes.io/projected/88de2dfb-9252-405d-b590-abf68ec6dbe9-kube-api-access-4dn98\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:43.138777 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:20:43.138738    5315 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88de2dfb_9252_405d_b590_abf68ec6dbe9.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 18:20:43.378105 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:43.378015    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-5fjng" event={"ID":"88de2dfb-9252-405d-b590-abf68ec6dbe9","Type":"ContainerDied","Data":"4a8ae3276073b0f6392a5c3509985779186ba1f0f39e6410dd823a22c4d09934"}
Nov 03 18:20:43.378105 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:43.378043    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-5fjng"
Nov 03 18:20:43.378105 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:20:43.378080    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a8ae3276073b0f6392a5c3509985779186ba1f0f39e6410dd823a22c4d09934"
Nov 03 18:20:54.576919 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:20:54.576875    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:21:01.623668 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:21:01.623632    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-97p65_c29d200a-c917-46e5-9cf5-21f4d952e470/ovn-acl-logging/0.log"
Nov 03 18:21:01.669770 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:21:01.669741    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-97p65_c29d200a-c917-46e5-9cf5-21f4d952e470/ovn-acl-logging/0.log"
Nov 03 18:21:05.372415 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:21:05.372380    5315 scope.go:122] "RemoveContainer" containerID="72dc22a814a81e21808ff59f3a138415350510459cd0c35b9e2058bdec025047"
Nov 03 18:21:05.381127 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:21:05.381102    5315 scope.go:122] "RemoveContainer" containerID="f0ad76a5fbeb9bf99b9de0f7adc53fa8e035349b19109278680f5276d8014fe3"
Nov 03 18:21:05.389442 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:21:05.389416    5315 scope.go:122] "RemoveContainer" containerID="7979bc31f2e3230cb6a459359c495abb5feb4ac9ac9820979a2d8149f86b7508"
Nov 03 18:21:05.397714 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:21:05.397686    5315 scope.go:122] "RemoveContainer" containerID="3a39b8e8c5e24ba7d67e14d0be941370f95cfdfe6965fe249f80ba249e1c15c3"
Nov 03 18:21:24.576305 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:21:24.576266    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:21:46.015198 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:21:46.015158    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tp6ht"]
Nov 03 18:21:46.015767 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:21:46.015728    5315 cpu_manager.go:414] "containerMap: removing container" podUID="88de2dfb-9252-405d-b590-abf68ec6dbe9" containerName="revoke-keys"
Nov 03 18:21:46.015767 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:21:46.015746    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88de2dfb-9252-405d-b590-abf68ec6dbe9" containerName="revoke-keys"
Nov 03 18:21:46.015866 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:21:46.015784    5315 cpu_manager.go:414] "containerMap: removing container" podUID="dd5c9fcf-e132-42e2-b031-c608cfd0b392" containerName="istio-proxy"
Nov 03 18:21:46.015866 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:21:46.015790    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd5c9fcf-e132-42e2-b031-c608cfd0b392" containerName="istio-proxy"
Nov 03 18:21:46.015866 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:21:46.015858    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="88de2dfb-9252-405d-b590-abf68ec6dbe9" containerName="revoke-keys"
Nov 03 18:21:46.019018 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:21:46.018997    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tp6ht"
Nov 03 18:21:46.067476 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:21:46.067438    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:21:46.067670 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:21:46.067494    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:21:46.067883 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:21:46.067859    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp8b6\" (UniqueName: \"kubernetes.io/projected/a5e59d9c-87a3-49d6-8efc-bb29567d789f-kube-api-access-pp8b6\") pod \"maas-api-revoke-sub-c1e271a9-tp6ht\" (UID: \"a5e59d9c-87a3-49d6-8efc-bb29567d789f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tp6ht"
Nov 03 18:21:46.067937 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:21:46.067920    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5e59d9c-87a3-49d6-8efc-bb29567d789f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-tp6ht\" (UID: \"a5e59d9c-87a3-49d6-8efc-bb29567d789f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tp6ht"
Nov 03 18:21:46.068030 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:21:46.068011    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5e59d9c-87a3-49d6-8efc-bb29567d789f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-tp6ht\" (UID: \"a5e59d9c-87a3-49d6-8efc-bb29567d789f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tp6ht"
Nov 03 18:21:46.115533 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:21:46.115500    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:21:46.115533 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:21:46.115525    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:21:46.117299 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:21:46.117271    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tp6ht"]
Nov 03 18:21:46.168627 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:21:46.168587    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp8b6\" (UniqueName: \"kubernetes.io/projected/a5e59d9c-87a3-49d6-8efc-bb29567d789f-kube-api-access-pp8b6\") pod \"maas-api-revoke-sub-c1e271a9-tp6ht\" (UID: \"a5e59d9c-87a3-49d6-8efc-bb29567d789f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tp6ht"
Nov 03 18:21:46.168787 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:21:46.168692    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5e59d9c-87a3-49d6-8efc-bb29567d789f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-tp6ht\" (UID: \"a5e59d9c-87a3-49d6-8efc-bb29567d789f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tp6ht"
Nov 03 18:21:46.168831 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:21:46.168784    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5e59d9c-87a3-49d6-8efc-bb29567d789f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-tp6ht\" (UID: \"a5e59d9c-87a3-49d6-8efc-bb29567d789f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tp6ht"
Nov 03 18:21:46.169314 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:21:46.169291    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5e59d9c-87a3-49d6-8efc-bb29567d789f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-tp6ht\" (UID: \"a5e59d9c-87a3-49d6-8efc-bb29567d789f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tp6ht"
Nov 03 18:21:46.171110 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:21:46.171091    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5e59d9c-87a3-49d6-8efc-bb29567d789f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-tp6ht\" (UID: \"a5e59d9c-87a3-49d6-8efc-bb29567d789f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tp6ht"
Nov 03 18:21:46.226348 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:21:46.226315    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp8b6\" (UniqueName: \"kubernetes.io/projected/a5e59d9c-87a3-49d6-8efc-bb29567d789f-kube-api-access-pp8b6\") pod \"maas-api-revoke-sub-c1e271a9-tp6ht\" (UID: \"a5e59d9c-87a3-49d6-8efc-bb29567d789f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tp6ht"
Nov 03 18:21:46.332207 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:21:46.332173    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tp6ht"
Nov 03 18:21:46.700018 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:21:46.699935    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tp6ht"]
Nov 03 18:21:46.706399 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:21:46.706366    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5e59d9c_87a3_49d6_8efc_bb29567d789f.slice/crio-8c3381f18faedd7abf49013122caa3d4fc423fe55cc151a7504634bb544f0618 WatchSource:0}: Error finding container 8c3381f18faedd7abf49013122caa3d4fc423fe55cc151a7504634bb544f0618: Status 404 returned error can't find the container with id 8c3381f18faedd7abf49013122caa3d4fc423fe55cc151a7504634bb544f0618
Nov 03 18:21:47.713464 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:21:47.713426    5315 generic.go:356] "Generic (PLEG): container finished" podID="a5e59d9c-87a3-49d6-8efc-bb29567d789f" containerID="f4ac6009772f327b65c2afdb1172cfb8c8c13abe2c9707f3b19276c5c1ed4669" exitCode=0
Nov 03 18:21:47.713931 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:21:47.713511    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tp6ht" event={"ID":"a5e59d9c-87a3-49d6-8efc-bb29567d789f","Type":"ContainerDied","Data":"f4ac6009772f327b65c2afdb1172cfb8c8c13abe2c9707f3b19276c5c1ed4669"}
Nov 03 18:21:47.713931 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:21:47.713640    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tp6ht" event={"ID":"a5e59d9c-87a3-49d6-8efc-bb29567d789f","Type":"ContainerStarted","Data":"8c3381f18faedd7abf49013122caa3d4fc423fe55cc151a7504634bb544f0618"}
Nov 03 18:21:48.849251 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:21:48.849226    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tp6ht"
Nov 03 18:21:48.894417 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:21:48.894391    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5e59d9c-87a3-49d6-8efc-bb29567d789f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5e59d9c-87a3-49d6-8efc-bb29567d789f-maas-api-openshift-ca\") pod \"a5e59d9c-87a3-49d6-8efc-bb29567d789f\" (UID: \"a5e59d9c-87a3-49d6-8efc-bb29567d789f\") "
Nov 03 18:21:48.894567 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:21:48.894465    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5e59d9c-87a3-49d6-8efc-bb29567d789f-kube-api-access-pp8b6\" (UniqueName: \"kubernetes.io/projected/a5e59d9c-87a3-49d6-8efc-bb29567d789f-kube-api-access-pp8b6\") pod \"a5e59d9c-87a3-49d6-8efc-bb29567d789f\" (UID: \"a5e59d9c-87a3-49d6-8efc-bb29567d789f\") "
Nov 03 18:21:48.894752 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:21:48.894731    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5e59d9c-87a3-49d6-8efc-bb29567d789f-maas-api-openshift-ca" pod "a5e59d9c-87a3-49d6-8efc-bb29567d789f" (UID: "a5e59d9c-87a3-49d6-8efc-bb29567d789f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:21:48.896448 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:21:48.896424    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5e59d9c-87a3-49d6-8efc-bb29567d789f-kube-api-access-pp8b6" pod "a5e59d9c-87a3-49d6-8efc-bb29567d789f" (UID: "a5e59d9c-87a3-49d6-8efc-bb29567d789f"). InnerVolumeSpecName "kube-api-access-pp8b6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:21:48.995036 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:21:48.994961    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5e59d9c-87a3-49d6-8efc-bb29567d789f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5e59d9c-87a3-49d6-8efc-bb29567d789f-maas-api-xks-ca\") pod \"a5e59d9c-87a3-49d6-8efc-bb29567d789f\" (UID: \"a5e59d9c-87a3-49d6-8efc-bb29567d789f\") "
Nov 03 18:21:48.995205 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:21:48.995190    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp8b6\" (UniqueName: \"kubernetes.io/projected/a5e59d9c-87a3-49d6-8efc-bb29567d789f-kube-api-access-pp8b6\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:21:48.995248 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:21:48.995207    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5e59d9c-87a3-49d6-8efc-bb29567d789f-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:21:48.996927 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:21:48.996897    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5e59d9c-87a3-49d6-8efc-bb29567d789f-maas-api-xks-ca" pod "a5e59d9c-87a3-49d6-8efc-bb29567d789f" (UID: "a5e59d9c-87a3-49d6-8efc-bb29567d789f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:21:49.095664 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:21:49.095600    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5e59d9c-87a3-49d6-8efc-bb29567d789f-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:21:49.726984 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:21:49.726942    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tp6ht" event={"ID":"a5e59d9c-87a3-49d6-8efc-bb29567d789f","Type":"ContainerDied","Data":"8c3381f18faedd7abf49013122caa3d4fc423fe55cc151a7504634bb544f0618"}
Nov 03 18:21:49.726984 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:21:49.726983    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tp6ht"
Nov 03 18:21:49.727202 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:21:49.727003    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c3381f18faedd7abf49013122caa3d4fc423fe55cc151a7504634bb544f0618"
Nov 03 18:21:54.576926 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:21:54.576881    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:22:05.443634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:05.442966    5315 scope.go:122] "RemoveContainer" containerID="314892e74754d6f2047867c109e23d6b10a000ed8782e15d545c2d9ab4996e53"
Nov 03 18:22:05.483890 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:05.483858    5315 scope.go:122] "RemoveContainer" containerID="101f5e7726e812113b1acb435f5bcc297b604944e7ebcd1c55de5c9e5523428c"
Nov 03 18:22:05.522300 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:05.522274    5315 scope.go:122] "RemoveContainer" containerID="12f308d10ef3d58e4bf4bab76a79145a186ecf2b8beb5c79f8c9b3bb7867aa02"
Nov 03 18:22:05.589744 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:05.581232    5315 scope.go:122] "RemoveContainer" containerID="a0e175c822312d8d7c282bd324796897a6deca6c690e94c82b0a4eb01ee5abfd"
Nov 03 18:22:05.637325 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:05.637210    5315 scope.go:122] "RemoveContainer" containerID="7b414d4fc8af67fe9c6688cd6fa371d101b514861b6370c6e3ab0aeceaf5c3c8"
Nov 03 18:22:05.676893 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:05.676849    5315 scope.go:122] "RemoveContainer" containerID="be6ab63213a6099f807323df01077cdaf4ca0d48196999f893b7511dc744b446"
Nov 03 18:22:14.045316 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:14.045235    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cac8e02-r44sw"]
Nov 03 18:22:14.091400 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:14.091365    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cac8e02-r44sw"]
Nov 03 18:22:14.092560 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:14.092541    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1183615-p4qqj"]
Nov 03 18:22:14.093636 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:14.093604    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1183615-p4qqj"]
Nov 03 18:22:15.102259 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:15.102226    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19759972-5257-430b-abed-7a2f601b652b" path="/var/lib/kubelet/pods/19759972-5257-430b-abed-7a2f601b652b/volumes"
Nov 03 18:22:15.102736 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:15.102719    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="394f9a52-30a5-4902-bba9-7e9442e7a67f" path="/var/lib/kubelet/pods/394f9a52-30a5-4902-bba9-7e9442e7a67f/volumes"
Nov 03 18:22:17.330805 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:17.330771    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7bg4x"]
Nov 03 18:22:17.331211 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:17.331160    5315 cpu_manager.go:414] "containerMap: removing container" podUID="a5e59d9c-87a3-49d6-8efc-bb29567d789f" containerName="revoke-keys"
Nov 03 18:22:17.331211 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:17.331171    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5e59d9c-87a3-49d6-8efc-bb29567d789f" containerName="revoke-keys"
Nov 03 18:22:17.331298 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:17.331241    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5e59d9c-87a3-49d6-8efc-bb29567d789f" containerName="revoke-keys"
Nov 03 18:22:17.336161 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:17.336136    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7bg4x"
Nov 03 18:22:17.384763 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:17.384722    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:22:17.413278 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:17.413246    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f9x7\" (UniqueName: \"kubernetes.io/projected/9094aa8c-6312-4f34-8636-a0b4a5c7c103-kube-api-access-5f9x7\") pod \"maas-api-revoke-sub-2e1f5b5f-7bg4x\" (UID: \"9094aa8c-6312-4f34-8636-a0b4a5c7c103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7bg4x"
Nov 03 18:22:17.413278 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:17.413289    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9094aa8c-6312-4f34-8636-a0b4a5c7c103-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-7bg4x\" (UID: \"9094aa8c-6312-4f34-8636-a0b4a5c7c103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7bg4x"
Nov 03 18:22:17.413558 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:17.413428    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9094aa8c-6312-4f34-8636-a0b4a5c7c103-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-7bg4x\" (UID: \"9094aa8c-6312-4f34-8636-a0b4a5c7c103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7bg4x"
Nov 03 18:22:17.430751 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:17.430720    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:22:17.430751 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:17.430738    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:22:17.430962 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:17.430812    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:22:17.432237 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:17.432211    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7bg4x"]
Nov 03 18:22:17.513969 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:17.513932    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9094aa8c-6312-4f34-8636-a0b4a5c7c103-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-7bg4x\" (UID: \"9094aa8c-6312-4f34-8636-a0b4a5c7c103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7bg4x"
Nov 03 18:22:17.514156 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:17.514020    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f9x7\" (UniqueName: \"kubernetes.io/projected/9094aa8c-6312-4f34-8636-a0b4a5c7c103-kube-api-access-5f9x7\") pod \"maas-api-revoke-sub-2e1f5b5f-7bg4x\" (UID: \"9094aa8c-6312-4f34-8636-a0b4a5c7c103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7bg4x"
Nov 03 18:22:17.514156 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:17.514059    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9094aa8c-6312-4f34-8636-a0b4a5c7c103-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-7bg4x\" (UID: \"9094aa8c-6312-4f34-8636-a0b4a5c7c103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7bg4x"
Nov 03 18:22:17.514764 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:17.514740    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9094aa8c-6312-4f34-8636-a0b4a5c7c103-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-7bg4x\" (UID: \"9094aa8c-6312-4f34-8636-a0b4a5c7c103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7bg4x"
Nov 03 18:22:17.516365 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:17.516345    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9094aa8c-6312-4f34-8636-a0b4a5c7c103-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-7bg4x\" (UID: \"9094aa8c-6312-4f34-8636-a0b4a5c7c103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7bg4x"
Nov 03 18:22:17.571742 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:17.571714    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f9x7\" (UniqueName: \"kubernetes.io/projected/9094aa8c-6312-4f34-8636-a0b4a5c7c103-kube-api-access-5f9x7\") pod \"maas-api-revoke-sub-2e1f5b5f-7bg4x\" (UID: \"9094aa8c-6312-4f34-8636-a0b4a5c7c103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7bg4x"
Nov 03 18:22:17.648302 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:17.648219    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7bg4x"
Nov 03 18:22:18.019522 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:18.019442    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7bg4x"]
Nov 03 18:22:18.026458 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:22:18.026430    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9094aa8c_6312_4f34_8636_a0b4a5c7c103.slice/crio-a06677d6105796ce332deb2aa15d2dd4aa7b6464916279aab8b7f51f3eef2686 WatchSource:0}: Error finding container a06677d6105796ce332deb2aa15d2dd4aa7b6464916279aab8b7f51f3eef2686: Status 404 returned error can't find the container with id a06677d6105796ce332deb2aa15d2dd4aa7b6464916279aab8b7f51f3eef2686
Nov 03 18:22:18.881623 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:18.881584    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7bg4x" event={"ID":"9094aa8c-6312-4f34-8636-a0b4a5c7c103","Type":"ContainerStarted","Data":"d36d7f02c8fecfbbc3e4403aff9882aa6a215cfea8fc6b3edd3bc6f5f794d124"}
Nov 03 18:22:18.882049 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:18.881661    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7bg4x" event={"ID":"9094aa8c-6312-4f34-8636-a0b4a5c7c103","Type":"ContainerStarted","Data":"a06677d6105796ce332deb2aa15d2dd4aa7b6464916279aab8b7f51f3eef2686"}
Nov 03 18:22:18.993977 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:18.993912    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7bg4x" podStartSLOduration=1.993892694 podStartE2EDuration="1.993892694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:22:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:22:18.992411275 +0000 UTC m=+2178.008609345" watchObservedRunningTime="2026-11-03 18:22:18.993892694 +0000 UTC m=+2178.010090765"
Nov 03 18:22:19.888363 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:19.888329    5315 generic.go:356] "Generic (PLEG): container finished" podID="9094aa8c-6312-4f34-8636-a0b4a5c7c103" containerID="d36d7f02c8fecfbbc3e4403aff9882aa6a215cfea8fc6b3edd3bc6f5f794d124" exitCode=0
Nov 03 18:22:19.888946 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:19.888416    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7bg4x" event={"ID":"9094aa8c-6312-4f34-8636-a0b4a5c7c103","Type":"ContainerDied","Data":"d36d7f02c8fecfbbc3e4403aff9882aa6a215cfea8fc6b3edd3bc6f5f794d124"}
Nov 03 18:22:21.027441 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:21.027412    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7bg4x"
Nov 03 18:22:21.148056 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:21.147981    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9094aa8c-6312-4f34-8636-a0b4a5c7c103-kube-api-access-5f9x7\" (UniqueName: \"kubernetes.io/projected/9094aa8c-6312-4f34-8636-a0b4a5c7c103-kube-api-access-5f9x7\") pod \"9094aa8c-6312-4f34-8636-a0b4a5c7c103\" (UID: \"9094aa8c-6312-4f34-8636-a0b4a5c7c103\") "
Nov 03 18:22:21.148211 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:21.148101    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9094aa8c-6312-4f34-8636-a0b4a5c7c103-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9094aa8c-6312-4f34-8636-a0b4a5c7c103-maas-api-xks-ca\") pod \"9094aa8c-6312-4f34-8636-a0b4a5c7c103\" (UID: \"9094aa8c-6312-4f34-8636-a0b4a5c7c103\") "
Nov 03 18:22:21.148211 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:21.148138    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9094aa8c-6312-4f34-8636-a0b4a5c7c103-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9094aa8c-6312-4f34-8636-a0b4a5c7c103-maas-api-openshift-ca\") pod \"9094aa8c-6312-4f34-8636-a0b4a5c7c103\" (UID: \"9094aa8c-6312-4f34-8636-a0b4a5c7c103\") "
Nov 03 18:22:21.148528 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:21.148503    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9094aa8c-6312-4f34-8636-a0b4a5c7c103-maas-api-openshift-ca" pod "9094aa8c-6312-4f34-8636-a0b4a5c7c103" (UID: "9094aa8c-6312-4f34-8636-a0b4a5c7c103"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:22:21.150461 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:21.150427    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9094aa8c-6312-4f34-8636-a0b4a5c7c103-maas-api-xks-ca" pod "9094aa8c-6312-4f34-8636-a0b4a5c7c103" (UID: "9094aa8c-6312-4f34-8636-a0b4a5c7c103"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:22:21.150568 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:21.150428    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9094aa8c-6312-4f34-8636-a0b4a5c7c103-kube-api-access-5f9x7" pod "9094aa8c-6312-4f34-8636-a0b4a5c7c103" (UID: "9094aa8c-6312-4f34-8636-a0b4a5c7c103"). InnerVolumeSpecName "kube-api-access-5f9x7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:22:21.249296 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:21.249253    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9094aa8c-6312-4f34-8636-a0b4a5c7c103-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:22:21.249296 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:21.249291    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9094aa8c-6312-4f34-8636-a0b4a5c7c103-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:22:21.249296 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:21.249303    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f9x7\" (UniqueName: \"kubernetes.io/projected/9094aa8c-6312-4f34-8636-a0b4a5c7c103-kube-api-access-5f9x7\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:22:21.898751 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:21.898716    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7bg4x"
Nov 03 18:22:21.898933 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:21.898759    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7bg4x" event={"ID":"9094aa8c-6312-4f34-8636-a0b4a5c7c103","Type":"ContainerDied","Data":"a06677d6105796ce332deb2aa15d2dd4aa7b6464916279aab8b7f51f3eef2686"}
Nov 03 18:22:21.898933 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:21.898807    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a06677d6105796ce332deb2aa15d2dd4aa7b6464916279aab8b7f51f3eef2686"
Nov 03 18:22:24.576408 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:22:24.576370    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:22:29.784940 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:29.784899    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7mt6j"]
Nov 03 18:22:29.785380 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:29.785313    5315 cpu_manager.go:414] "containerMap: removing container" podUID="9094aa8c-6312-4f34-8636-a0b4a5c7c103" containerName="revoke-keys"
Nov 03 18:22:29.785380 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:29.785326    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9094aa8c-6312-4f34-8636-a0b4a5c7c103" containerName="revoke-keys"
Nov 03 18:22:29.785459 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:29.785390    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="9094aa8c-6312-4f34-8636-a0b4a5c7c103" containerName="revoke-keys"
Nov 03 18:22:29.788562 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:29.788539    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7mt6j"
Nov 03 18:22:29.837316 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:29.837286    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:22:29.883420 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:29.883382    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:22:29.883420 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:29.883406    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:22:29.883420 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:29.883393    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:22:29.885006 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:29.884984    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7mt6j"]
Nov 03 18:22:29.931762 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:29.931734    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce4b89d6-a516-4513-8a2a-6e04be561d17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-7mt6j\" (UID: \"ce4b89d6-a516-4513-8a2a-6e04be561d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7mt6j"
Nov 03 18:22:29.931762 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:29.931768    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df2xp\" (UniqueName: \"kubernetes.io/projected/ce4b89d6-a516-4513-8a2a-6e04be561d17-kube-api-access-df2xp\") pod \"maas-api-revoke-sub-ec843018-7mt6j\" (UID: \"ce4b89d6-a516-4513-8a2a-6e04be561d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7mt6j"
Nov 03 18:22:29.931997 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:29.931832    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce4b89d6-a516-4513-8a2a-6e04be561d17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-7mt6j\" (UID: \"ce4b89d6-a516-4513-8a2a-6e04be561d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7mt6j"
Nov 03 18:22:30.033012 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:30.032977    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df2xp\" (UniqueName: \"kubernetes.io/projected/ce4b89d6-a516-4513-8a2a-6e04be561d17-kube-api-access-df2xp\") pod \"maas-api-revoke-sub-ec843018-7mt6j\" (UID: \"ce4b89d6-a516-4513-8a2a-6e04be561d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7mt6j"
Nov 03 18:22:30.033212 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:30.033029    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce4b89d6-a516-4513-8a2a-6e04be561d17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-7mt6j\" (UID: \"ce4b89d6-a516-4513-8a2a-6e04be561d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7mt6j"
Nov 03 18:22:30.033212 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:30.033117    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce4b89d6-a516-4513-8a2a-6e04be561d17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-7mt6j\" (UID: \"ce4b89d6-a516-4513-8a2a-6e04be561d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7mt6j"
Nov 03 18:22:30.033719 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:30.033696    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce4b89d6-a516-4513-8a2a-6e04be561d17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-7mt6j\" (UID: \"ce4b89d6-a516-4513-8a2a-6e04be561d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7mt6j"
Nov 03 18:22:30.035654 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:30.035562    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce4b89d6-a516-4513-8a2a-6e04be561d17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-7mt6j\" (UID: \"ce4b89d6-a516-4513-8a2a-6e04be561d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7mt6j"
Nov 03 18:22:30.090733 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:30.090701    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df2xp\" (UniqueName: \"kubernetes.io/projected/ce4b89d6-a516-4513-8a2a-6e04be561d17-kube-api-access-df2xp\") pod \"maas-api-revoke-sub-ec843018-7mt6j\" (UID: \"ce4b89d6-a516-4513-8a2a-6e04be561d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7mt6j"
Nov 03 18:22:30.101577 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:30.101554    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7mt6j"
Nov 03 18:22:30.469483 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:30.469384    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7mt6j"]
Nov 03 18:22:30.476337 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:22:30.476302    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce4b89d6_a516_4513_8a2a_6e04be561d17.slice/crio-28c902d7e11c0a30f7eb7cbdcb79e54be108f64013e059da260a98669360928d WatchSource:0}: Error finding container 28c902d7e11c0a30f7eb7cbdcb79e54be108f64013e059da260a98669360928d: Status 404 returned error can't find the container with id 28c902d7e11c0a30f7eb7cbdcb79e54be108f64013e059da260a98669360928d
Nov 03 18:22:30.944339 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:30.944306    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7mt6j" event={"ID":"ce4b89d6-a516-4513-8a2a-6e04be561d17","Type":"ContainerStarted","Data":"8d4ded2b6669cd2e9d6336156a226f44b0038c14a649d7a87638238842721503"}
Nov 03 18:22:30.944755 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:30.944362    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7mt6j" event={"ID":"ce4b89d6-a516-4513-8a2a-6e04be561d17","Type":"ContainerStarted","Data":"28c902d7e11c0a30f7eb7cbdcb79e54be108f64013e059da260a98669360928d"}
Nov 03 18:22:31.053187 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:31.053137    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7mt6j" podStartSLOduration=2.053122807 podStartE2EDuration="2.053122807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:22:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:22:31.051475784 +0000 UTC m=+2190.067673853" watchObservedRunningTime="2026-11-03 18:22:31.053122807 +0000 UTC m=+2190.069320872"
Nov 03 18:22:31.950788 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:31.950750    5315 generic.go:356] "Generic (PLEG): container finished" podID="ce4b89d6-a516-4513-8a2a-6e04be561d17" containerID="8d4ded2b6669cd2e9d6336156a226f44b0038c14a649d7a87638238842721503" exitCode=0
Nov 03 18:22:31.951185 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:31.950797    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7mt6j" event={"ID":"ce4b89d6-a516-4513-8a2a-6e04be561d17","Type":"ContainerDied","Data":"8d4ded2b6669cd2e9d6336156a226f44b0038c14a649d7a87638238842721503"}
Nov 03 18:22:33.091938 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:33.091909    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7mt6j"
Nov 03 18:22:33.256669 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:33.256560    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce4b89d6-a516-4513-8a2a-6e04be561d17-kube-api-access-df2xp\" (UniqueName: \"kubernetes.io/projected/ce4b89d6-a516-4513-8a2a-6e04be561d17-kube-api-access-df2xp\") pod \"ce4b89d6-a516-4513-8a2a-6e04be561d17\" (UID: \"ce4b89d6-a516-4513-8a2a-6e04be561d17\") "
Nov 03 18:22:33.256829 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:33.256675    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce4b89d6-a516-4513-8a2a-6e04be561d17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce4b89d6-a516-4513-8a2a-6e04be561d17-maas-api-openshift-ca\") pod \"ce4b89d6-a516-4513-8a2a-6e04be561d17\" (UID: \"ce4b89d6-a516-4513-8a2a-6e04be561d17\") "
Nov 03 18:22:33.256829 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:33.256746    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce4b89d6-a516-4513-8a2a-6e04be561d17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce4b89d6-a516-4513-8a2a-6e04be561d17-maas-api-xks-ca\") pod \"ce4b89d6-a516-4513-8a2a-6e04be561d17\" (UID: \"ce4b89d6-a516-4513-8a2a-6e04be561d17\") "
Nov 03 18:22:33.257047 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:33.257016    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce4b89d6-a516-4513-8a2a-6e04be561d17-maas-api-openshift-ca" pod "ce4b89d6-a516-4513-8a2a-6e04be561d17" (UID: "ce4b89d6-a516-4513-8a2a-6e04be561d17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:22:33.258813 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:33.258787    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce4b89d6-a516-4513-8a2a-6e04be561d17-kube-api-access-df2xp" pod "ce4b89d6-a516-4513-8a2a-6e04be561d17" (UID: "ce4b89d6-a516-4513-8a2a-6e04be561d17"). InnerVolumeSpecName "kube-api-access-df2xp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:22:33.258902 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:33.258826    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce4b89d6-a516-4513-8a2a-6e04be561d17-maas-api-xks-ca" pod "ce4b89d6-a516-4513-8a2a-6e04be561d17" (UID: "ce4b89d6-a516-4513-8a2a-6e04be561d17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:22:33.357822 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:33.357784    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce4b89d6-a516-4513-8a2a-6e04be561d17-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:22:33.357822 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:33.357815    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce4b89d6-a516-4513-8a2a-6e04be561d17-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:22:33.357822 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:33.357826    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df2xp\" (UniqueName: \"kubernetes.io/projected/ce4b89d6-a516-4513-8a2a-6e04be561d17-kube-api-access-df2xp\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:22:33.962043 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:33.962004    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7mt6j" event={"ID":"ce4b89d6-a516-4513-8a2a-6e04be561d17","Type":"ContainerDied","Data":"28c902d7e11c0a30f7eb7cbdcb79e54be108f64013e059da260a98669360928d"}
Nov 03 18:22:33.962043 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:33.962041    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7mt6j"
Nov 03 18:22:33.962265 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:33.962066    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28c902d7e11c0a30f7eb7cbdcb79e54be108f64013e059da260a98669360928d"
Nov 03 18:22:40.046666 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:40.046631    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8941c20-8kr5x"]
Nov 03 18:22:40.092446 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:40.092412    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8941c20-8kr5x"]
Nov 03 18:22:41.045224 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:41.045189    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b04d452e-wlv5h"]
Nov 03 18:22:41.090927 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:41.090895    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b04d452e-wlv5h"]
Nov 03 18:22:41.102431 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:41.102397    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91b95076-e971-49d4-b183-66e0da518d56" path="/var/lib/kubelet/pods/91b95076-e971-49d4-b183-66e0da518d56/volumes"
Nov 03 18:22:41.102916 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:22:41.102893    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e376e147-b415-47d0-8bb8-21aecf441da9" path="/var/lib/kubelet/pods/e376e147-b415-47d0-8bb8-21aecf441da9/volumes"
Nov 03 18:22:54.577484 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:22:54.577441    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:23:05.406314 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:05.406275    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vgw55"]
Nov 03 18:23:05.406765 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:05.406624    5315 cpu_manager.go:414] "containerMap: removing container" podUID="ce4b89d6-a516-4513-8a2a-6e04be561d17" containerName="revoke-keys"
Nov 03 18:23:05.406765 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:05.406636    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce4b89d6-a516-4513-8a2a-6e04be561d17" containerName="revoke-keys"
Nov 03 18:23:05.406765 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:05.406716    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce4b89d6-a516-4513-8a2a-6e04be561d17" containerName="revoke-keys"
Nov 03 18:23:05.409930 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:05.409906    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vgw55"
Nov 03 18:23:05.458348 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:05.458318    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:23:05.475903 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:05.475864    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs8dt\" (UniqueName: \"kubernetes.io/projected/0625122c-faa5-4942-b9e3-c764b52e4a1b-kube-api-access-zs8dt\") pod \"maas-api-revoke-sub-046e1343-vgw55\" (UID: \"0625122c-faa5-4942-b9e3-c764b52e4a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vgw55"
Nov 03 18:23:05.476048 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:05.475920    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0625122c-faa5-4942-b9e3-c764b52e4a1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-vgw55\" (UID: \"0625122c-faa5-4942-b9e3-c764b52e4a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vgw55"
Nov 03 18:23:05.476048 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:05.476022    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0625122c-faa5-4942-b9e3-c764b52e4a1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-vgw55\" (UID: \"0625122c-faa5-4942-b9e3-c764b52e4a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vgw55"
Nov 03 18:23:05.504602 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:05.504567    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:23:05.504602 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:05.504575    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:23:05.504852 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:05.504676    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:23:05.506246 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:05.506222    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vgw55"]
Nov 03 18:23:05.576717 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:05.576684    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0625122c-faa5-4942-b9e3-c764b52e4a1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-vgw55\" (UID: \"0625122c-faa5-4942-b9e3-c764b52e4a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vgw55"
Nov 03 18:23:05.576938 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:05.576792    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs8dt\" (UniqueName: \"kubernetes.io/projected/0625122c-faa5-4942-b9e3-c764b52e4a1b-kube-api-access-zs8dt\") pod \"maas-api-revoke-sub-046e1343-vgw55\" (UID: \"0625122c-faa5-4942-b9e3-c764b52e4a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vgw55"
Nov 03 18:23:05.576938 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:05.576841    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0625122c-faa5-4942-b9e3-c764b52e4a1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-vgw55\" (UID: \"0625122c-faa5-4942-b9e3-c764b52e4a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vgw55"
Nov 03 18:23:05.577356 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:05.577333    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0625122c-faa5-4942-b9e3-c764b52e4a1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-vgw55\" (UID: \"0625122c-faa5-4942-b9e3-c764b52e4a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vgw55"
Nov 03 18:23:05.579139 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:05.579116    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0625122c-faa5-4942-b9e3-c764b52e4a1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-vgw55\" (UID: \"0625122c-faa5-4942-b9e3-c764b52e4a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vgw55"
Nov 03 18:23:05.633757 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:05.633724    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs8dt\" (UniqueName: \"kubernetes.io/projected/0625122c-faa5-4942-b9e3-c764b52e4a1b-kube-api-access-zs8dt\") pod \"maas-api-revoke-sub-046e1343-vgw55\" (UID: \"0625122c-faa5-4942-b9e3-c764b52e4a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vgw55"
Nov 03 18:23:05.722442 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:05.722357    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vgw55"
Nov 03 18:23:05.773282 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:05.772764    5315 scope.go:122] "RemoveContainer" containerID="8d52257469951c2068c5f5823a43128fb9db800497f04b6c1fcf43586b8a81de"
Nov 03 18:23:05.786969 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:05.786941    5315 scope.go:122] "RemoveContainer" containerID="0cd309e994a848d938ba0e26fa9cecf0fb5a0cdfc3858e0bf62a0d6d9c7ffbd7"
Nov 03 18:23:06.092976 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:06.092939    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vgw55"]
Nov 03 18:23:06.101166 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:23:06.101131    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0625122c_faa5_4942_b9e3_c764b52e4a1b.slice/crio-d62d4d08022acb8db902a337bb861526fb05bd67b9273b646533c0815f226d98 WatchSource:0}: Error finding container d62d4d08022acb8db902a337bb861526fb05bd67b9273b646533c0815f226d98: Status 404 returned error can't find the container with id d62d4d08022acb8db902a337bb861526fb05bd67b9273b646533c0815f226d98
Nov 03 18:23:06.150645 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:06.150599    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vgw55" event={"ID":"0625122c-faa5-4942-b9e3-c764b52e4a1b","Type":"ContainerStarted","Data":"d62d4d08022acb8db902a337bb861526fb05bd67b9273b646533c0815f226d98"}
Nov 03 18:23:07.156788 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:07.156752    5315 generic.go:356] "Generic (PLEG): container finished" podID="0625122c-faa5-4942-b9e3-c764b52e4a1b" containerID="ea6f8e047e20ac33836dcbf00a00460d1cfe6b5808bfabe0464a8f8f2726c82a" exitCode=0
Nov 03 18:23:07.157128 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:07.156801    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vgw55" event={"ID":"0625122c-faa5-4942-b9e3-c764b52e4a1b","Type":"ContainerDied","Data":"ea6f8e047e20ac33836dcbf00a00460d1cfe6b5808bfabe0464a8f8f2726c82a"}
Nov 03 18:23:08.292111 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:08.292085    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vgw55"
Nov 03 18:23:08.406937 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:08.406906    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0625122c-faa5-4942-b9e3-c764b52e4a1b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0625122c-faa5-4942-b9e3-c764b52e4a1b-maas-api-openshift-ca\") pod \"0625122c-faa5-4942-b9e3-c764b52e4a1b\" (UID: \"0625122c-faa5-4942-b9e3-c764b52e4a1b\") "
Nov 03 18:23:08.407161 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:08.406957    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0625122c-faa5-4942-b9e3-c764b52e4a1b-kube-api-access-zs8dt\" (UniqueName: \"kubernetes.io/projected/0625122c-faa5-4942-b9e3-c764b52e4a1b-kube-api-access-zs8dt\") pod \"0625122c-faa5-4942-b9e3-c764b52e4a1b\" (UID: \"0625122c-faa5-4942-b9e3-c764b52e4a1b\") "
Nov 03 18:23:08.407161 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:08.407022    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0625122c-faa5-4942-b9e3-c764b52e4a1b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0625122c-faa5-4942-b9e3-c764b52e4a1b-maas-api-xks-ca\") pod \"0625122c-faa5-4942-b9e3-c764b52e4a1b\" (UID: \"0625122c-faa5-4942-b9e3-c764b52e4a1b\") "
Nov 03 18:23:08.407344 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:08.407316    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0625122c-faa5-4942-b9e3-c764b52e4a1b-maas-api-openshift-ca" pod "0625122c-faa5-4942-b9e3-c764b52e4a1b" (UID: "0625122c-faa5-4942-b9e3-c764b52e4a1b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:23:08.411035 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:08.411010    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0625122c-faa5-4942-b9e3-c764b52e4a1b-kube-api-access-zs8dt" pod "0625122c-faa5-4942-b9e3-c764b52e4a1b" (UID: "0625122c-faa5-4942-b9e3-c764b52e4a1b"). InnerVolumeSpecName "kube-api-access-zs8dt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:23:08.411139 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:08.411009    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0625122c-faa5-4942-b9e3-c764b52e4a1b-maas-api-xks-ca" pod "0625122c-faa5-4942-b9e3-c764b52e4a1b" (UID: "0625122c-faa5-4942-b9e3-c764b52e4a1b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:23:08.508347 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:08.508306    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0625122c-faa5-4942-b9e3-c764b52e4a1b-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:23:08.508347 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:08.508346    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs8dt\" (UniqueName: \"kubernetes.io/projected/0625122c-faa5-4942-b9e3-c764b52e4a1b-kube-api-access-zs8dt\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:23:08.508559 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:08.508360    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0625122c-faa5-4942-b9e3-c764b52e4a1b-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:23:09.166963 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:09.166927    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vgw55"
Nov 03 18:23:09.167154 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:09.166920    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vgw55" event={"ID":"0625122c-faa5-4942-b9e3-c764b52e4a1b","Type":"ContainerDied","Data":"d62d4d08022acb8db902a337bb861526fb05bd67b9273b646533c0815f226d98"}
Nov 03 18:23:09.167154 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:09.167065    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d62d4d08022acb8db902a337bb861526fb05bd67b9273b646533c0815f226d98"
Nov 03 18:23:16.216881 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:16.216834    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wzbdd"]
Nov 03 18:23:16.217327 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:16.217185    5315 cpu_manager.go:414] "containerMap: removing container" podUID="0625122c-faa5-4942-b9e3-c764b52e4a1b" containerName="revoke-keys"
Nov 03 18:23:16.217327 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:16.217195    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0625122c-faa5-4942-b9e3-c764b52e4a1b" containerName="revoke-keys"
Nov 03 18:23:16.217327 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:16.217281    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="0625122c-faa5-4942-b9e3-c764b52e4a1b" containerName="revoke-keys"
Nov 03 18:23:16.220469 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:16.220448    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wzbdd"
Nov 03 18:23:16.269241 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:16.269207    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:23:16.269408 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:16.269237    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:23:16.315108 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:16.315076    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:23:16.315318 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:16.315299    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:23:16.317033 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:16.317009    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wzbdd"]
Nov 03 18:23:16.383684 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:16.383648    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h7kz\" (UniqueName: \"kubernetes.io/projected/a29bc978-93c9-4602-88c3-ceefba86050b-kube-api-access-7h7kz\") pod \"maas-api-revoke-sub-d3fba056-wzbdd\" (UID: \"a29bc978-93c9-4602-88c3-ceefba86050b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wzbdd"
Nov 03 18:23:16.383856 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:16.383767    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29bc978-93c9-4602-88c3-ceefba86050b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-wzbdd\" (UID: \"a29bc978-93c9-4602-88c3-ceefba86050b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wzbdd"
Nov 03 18:23:16.383856 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:16.383809    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29bc978-93c9-4602-88c3-ceefba86050b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-wzbdd\" (UID: \"a29bc978-93c9-4602-88c3-ceefba86050b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wzbdd"
Nov 03 18:23:16.484875 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:16.484784    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29bc978-93c9-4602-88c3-ceefba86050b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-wzbdd\" (UID: \"a29bc978-93c9-4602-88c3-ceefba86050b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wzbdd"
Nov 03 18:23:16.484875 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:16.484852    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h7kz\" (UniqueName: \"kubernetes.io/projected/a29bc978-93c9-4602-88c3-ceefba86050b-kube-api-access-7h7kz\") pod \"maas-api-revoke-sub-d3fba056-wzbdd\" (UID: \"a29bc978-93c9-4602-88c3-ceefba86050b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wzbdd"
Nov 03 18:23:16.485068 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:16.484897    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29bc978-93c9-4602-88c3-ceefba86050b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-wzbdd\" (UID: \"a29bc978-93c9-4602-88c3-ceefba86050b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wzbdd"
Nov 03 18:23:16.485567 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:16.485546    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29bc978-93c9-4602-88c3-ceefba86050b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-wzbdd\" (UID: \"a29bc978-93c9-4602-88c3-ceefba86050b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wzbdd"
Nov 03 18:23:16.487257 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:16.487234    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29bc978-93c9-4602-88c3-ceefba86050b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-wzbdd\" (UID: \"a29bc978-93c9-4602-88c3-ceefba86050b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wzbdd"
Nov 03 18:23:16.542292 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:16.542262    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h7kz\" (UniqueName: \"kubernetes.io/projected/a29bc978-93c9-4602-88c3-ceefba86050b-kube-api-access-7h7kz\") pod \"maas-api-revoke-sub-d3fba056-wzbdd\" (UID: \"a29bc978-93c9-4602-88c3-ceefba86050b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wzbdd"
Nov 03 18:23:16.833336 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:16.833302    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wzbdd"
Nov 03 18:23:17.205201 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:17.205107    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wzbdd"]
Nov 03 18:23:17.211534 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:23:17.211501    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda29bc978_93c9_4602_88c3_ceefba86050b.slice/crio-be625f9753ac19a803250f4cf01ce767607de19d7703de2aab55471edec97362 WatchSource:0}: Error finding container be625f9753ac19a803250f4cf01ce767607de19d7703de2aab55471edec97362: Status 404 returned error can't find the container with id be625f9753ac19a803250f4cf01ce767607de19d7703de2aab55471edec97362
Nov 03 18:23:18.213957 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:18.213924    5315 generic.go:356] "Generic (PLEG): container finished" podID="a29bc978-93c9-4602-88c3-ceefba86050b" containerID="35b2bc8e1b347aa6d78b80e101e5e0553e62905df07f7e9cfeec705b7c6ec4d4" exitCode=0
Nov 03 18:23:18.214340 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:18.214010    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wzbdd" event={"ID":"a29bc978-93c9-4602-88c3-ceefba86050b","Type":"ContainerDied","Data":"35b2bc8e1b347aa6d78b80e101e5e0553e62905df07f7e9cfeec705b7c6ec4d4"}
Nov 03 18:23:18.214340 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:18.214067    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wzbdd" event={"ID":"a29bc978-93c9-4602-88c3-ceefba86050b","Type":"ContainerStarted","Data":"be625f9753ac19a803250f4cf01ce767607de19d7703de2aab55471edec97362"}
Nov 03 18:23:19.414555 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:19.414517    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wzbdd"
Nov 03 18:23:19.418636 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:19.418587    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a29bc978-93c9-4602-88c3-ceefba86050b-kube-api-access-7h7kz\" (UniqueName: \"kubernetes.io/projected/a29bc978-93c9-4602-88c3-ceefba86050b-kube-api-access-7h7kz\") pod \"a29bc978-93c9-4602-88c3-ceefba86050b\" (UID: \"a29bc978-93c9-4602-88c3-ceefba86050b\") "
Nov 03 18:23:19.418790 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:19.418666    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a29bc978-93c9-4602-88c3-ceefba86050b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29bc978-93c9-4602-88c3-ceefba86050b-maas-api-openshift-ca\") pod \"a29bc978-93c9-4602-88c3-ceefba86050b\" (UID: \"a29bc978-93c9-4602-88c3-ceefba86050b\") "
Nov 03 18:23:19.418790 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:19.418722    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a29bc978-93c9-4602-88c3-ceefba86050b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29bc978-93c9-4602-88c3-ceefba86050b-maas-api-xks-ca\") pod \"a29bc978-93c9-4602-88c3-ceefba86050b\" (UID: \"a29bc978-93c9-4602-88c3-ceefba86050b\") "
Nov 03 18:23:19.419384 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:19.419315    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a29bc978-93c9-4602-88c3-ceefba86050b-maas-api-openshift-ca" pod "a29bc978-93c9-4602-88c3-ceefba86050b" (UID: "a29bc978-93c9-4602-88c3-ceefba86050b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:23:19.421507 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:19.421454    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a29bc978-93c9-4602-88c3-ceefba86050b-maas-api-xks-ca" pod "a29bc978-93c9-4602-88c3-ceefba86050b" (UID: "a29bc978-93c9-4602-88c3-ceefba86050b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:23:19.421507 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:19.421459    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a29bc978-93c9-4602-88c3-ceefba86050b-kube-api-access-7h7kz" pod "a29bc978-93c9-4602-88c3-ceefba86050b" (UID: "a29bc978-93c9-4602-88c3-ceefba86050b"). InnerVolumeSpecName "kube-api-access-7h7kz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:23:19.522532 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:19.522207    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h7kz\" (UniqueName: \"kubernetes.io/projected/a29bc978-93c9-4602-88c3-ceefba86050b-kube-api-access-7h7kz\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:23:19.522532 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:19.522263    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29bc978-93c9-4602-88c3-ceefba86050b-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:23:19.522532 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:19.522281    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29bc978-93c9-4602-88c3-ceefba86050b-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:23:20.256646 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:20.246206    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wzbdd" event={"ID":"a29bc978-93c9-4602-88c3-ceefba86050b","Type":"ContainerDied","Data":"be625f9753ac19a803250f4cf01ce767607de19d7703de2aab55471edec97362"}
Nov 03 18:23:20.256646 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:20.246276    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be625f9753ac19a803250f4cf01ce767607de19d7703de2aab55471edec97362"
Nov 03 18:23:20.256646 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:20.246346    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wzbdd"
Nov 03 18:23:24.577109 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:23:24.577072    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:23:51.148813 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:51.148766    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wqs8h"]
Nov 03 18:23:51.149334 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:51.149304    5315 cpu_manager.go:414] "containerMap: removing container" podUID="a29bc978-93c9-4602-88c3-ceefba86050b" containerName="revoke-keys"
Nov 03 18:23:51.149334 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:51.149323    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a29bc978-93c9-4602-88c3-ceefba86050b" containerName="revoke-keys"
Nov 03 18:23:51.149455 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:51.149432    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="a29bc978-93c9-4602-88c3-ceefba86050b" containerName="revoke-keys"
Nov 03 18:23:51.152641 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:51.152601    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wqs8h"
Nov 03 18:23:51.201488 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:51.201458    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:23:51.236765 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:51.236728    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e656d6f-9d30-4600-b4ac-5fa8aa0cade2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-wqs8h\" (UID: \"2e656d6f-9d30-4600-b4ac-5fa8aa0cade2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wqs8h"
Nov 03 18:23:51.236932 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:51.236775    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qk9k\" (UniqueName: \"kubernetes.io/projected/2e656d6f-9d30-4600-b4ac-5fa8aa0cade2-kube-api-access-8qk9k\") pod \"maas-api-revoke-sub-ae594979-wqs8h\" (UID: \"2e656d6f-9d30-4600-b4ac-5fa8aa0cade2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wqs8h"
Nov 03 18:23:51.236932 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:51.236794    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e656d6f-9d30-4600-b4ac-5fa8aa0cade2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-wqs8h\" (UID: \"2e656d6f-9d30-4600-b4ac-5fa8aa0cade2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wqs8h"
Nov 03 18:23:51.247653 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:51.247601    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:23:51.247653 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:51.247636    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:23:51.247836 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:51.247671    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:23:51.249408 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:51.249386    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wqs8h"]
Nov 03 18:23:51.338095 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:51.338049    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e656d6f-9d30-4600-b4ac-5fa8aa0cade2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-wqs8h\" (UID: \"2e656d6f-9d30-4600-b4ac-5fa8aa0cade2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wqs8h"
Nov 03 18:23:51.338415 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:51.338110    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qk9k\" (UniqueName: \"kubernetes.io/projected/2e656d6f-9d30-4600-b4ac-5fa8aa0cade2-kube-api-access-8qk9k\") pod \"maas-api-revoke-sub-ae594979-wqs8h\" (UID: \"2e656d6f-9d30-4600-b4ac-5fa8aa0cade2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wqs8h"
Nov 03 18:23:51.338415 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:51.338149    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e656d6f-9d30-4600-b4ac-5fa8aa0cade2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-wqs8h\" (UID: \"2e656d6f-9d30-4600-b4ac-5fa8aa0cade2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wqs8h"
Nov 03 18:23:51.338969 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:51.338942    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e656d6f-9d30-4600-b4ac-5fa8aa0cade2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-wqs8h\" (UID: \"2e656d6f-9d30-4600-b4ac-5fa8aa0cade2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wqs8h"
Nov 03 18:23:51.340442 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:51.340418    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e656d6f-9d30-4600-b4ac-5fa8aa0cade2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-wqs8h\" (UID: \"2e656d6f-9d30-4600-b4ac-5fa8aa0cade2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wqs8h"
Nov 03 18:23:51.395640 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:51.395599    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qk9k\" (UniqueName: \"kubernetes.io/projected/2e656d6f-9d30-4600-b4ac-5fa8aa0cade2-kube-api-access-8qk9k\") pod \"maas-api-revoke-sub-ae594979-wqs8h\" (UID: \"2e656d6f-9d30-4600-b4ac-5fa8aa0cade2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wqs8h"
Nov 03 18:23:51.465905 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:51.465829    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wqs8h"
Nov 03 18:23:51.832639 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:51.832595    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wqs8h"]
Nov 03 18:23:51.839511 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:23:51.839483    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e656d6f_9d30_4600_b4ac_5fa8aa0cade2.slice/crio-dbbb2a01500e750c9cd0820bbd733efcc4d7ecf95ea0aaa825f0499ae79c225f WatchSource:0}: Error finding container dbbb2a01500e750c9cd0820bbd733efcc4d7ecf95ea0aaa825f0499ae79c225f: Status 404 returned error can't find the container with id dbbb2a01500e750c9cd0820bbd733efcc4d7ecf95ea0aaa825f0499ae79c225f
Nov 03 18:23:52.445047 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:52.444946    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wqs8h" event={"ID":"2e656d6f-9d30-4600-b4ac-5fa8aa0cade2","Type":"ContainerStarted","Data":"9603f8f33344f6369e614279a3734dfe216411fc716fa4150b96c77310251897"}
Nov 03 18:23:52.445047 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:52.445014    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wqs8h" event={"ID":"2e656d6f-9d30-4600-b4ac-5fa8aa0cade2","Type":"ContainerStarted","Data":"dbbb2a01500e750c9cd0820bbd733efcc4d7ecf95ea0aaa825f0499ae79c225f"}
Nov 03 18:23:52.553829 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:52.553771    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wqs8h" podStartSLOduration=1.553757118 podStartE2EDuration="1.553757118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:23:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:23:52.552683519 +0000 UTC m=+2271.568881581" watchObservedRunningTime="2026-11-03 18:23:52.553757118 +0000 UTC m=+2271.569955186"
Nov 03 18:23:53.451647 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:53.451589    5315 generic.go:356] "Generic (PLEG): container finished" podID="2e656d6f-9d30-4600-b4ac-5fa8aa0cade2" containerID="9603f8f33344f6369e614279a3734dfe216411fc716fa4150b96c77310251897" exitCode=0
Nov 03 18:23:53.452061 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:53.451679    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wqs8h" event={"ID":"2e656d6f-9d30-4600-b4ac-5fa8aa0cade2","Type":"ContainerDied","Data":"9603f8f33344f6369e614279a3734dfe216411fc716fa4150b96c77310251897"}
Nov 03 18:23:54.582292 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:23:54.582255    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:23:54.609214 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:54.609190    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wqs8h"
Nov 03 18:23:54.669069 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:54.669039    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e656d6f-9d30-4600-b4ac-5fa8aa0cade2-kube-api-access-8qk9k\" (UniqueName: \"kubernetes.io/projected/2e656d6f-9d30-4600-b4ac-5fa8aa0cade2-kube-api-access-8qk9k\") pod \"2e656d6f-9d30-4600-b4ac-5fa8aa0cade2\" (UID: \"2e656d6f-9d30-4600-b4ac-5fa8aa0cade2\") "
Nov 03 18:23:54.669215 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:54.669085    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e656d6f-9d30-4600-b4ac-5fa8aa0cade2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e656d6f-9d30-4600-b4ac-5fa8aa0cade2-maas-api-openshift-ca\") pod \"2e656d6f-9d30-4600-b4ac-5fa8aa0cade2\" (UID: \"2e656d6f-9d30-4600-b4ac-5fa8aa0cade2\") "
Nov 03 18:23:54.669215 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:54.669201    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e656d6f-9d30-4600-b4ac-5fa8aa0cade2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e656d6f-9d30-4600-b4ac-5fa8aa0cade2-maas-api-xks-ca\") pod \"2e656d6f-9d30-4600-b4ac-5fa8aa0cade2\" (UID: \"2e656d6f-9d30-4600-b4ac-5fa8aa0cade2\") "
Nov 03 18:23:54.669479 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:54.669456    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e656d6f-9d30-4600-b4ac-5fa8aa0cade2-maas-api-openshift-ca" pod "2e656d6f-9d30-4600-b4ac-5fa8aa0cade2" (UID: "2e656d6f-9d30-4600-b4ac-5fa8aa0cade2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:23:54.671204 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:54.671182    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e656d6f-9d30-4600-b4ac-5fa8aa0cade2-maas-api-xks-ca" pod "2e656d6f-9d30-4600-b4ac-5fa8aa0cade2" (UID: "2e656d6f-9d30-4600-b4ac-5fa8aa0cade2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:23:54.671283 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:54.671217    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e656d6f-9d30-4600-b4ac-5fa8aa0cade2-kube-api-access-8qk9k" pod "2e656d6f-9d30-4600-b4ac-5fa8aa0cade2" (UID: "2e656d6f-9d30-4600-b4ac-5fa8aa0cade2"). InnerVolumeSpecName "kube-api-access-8qk9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:23:54.771057 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:54.770968    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e656d6f-9d30-4600-b4ac-5fa8aa0cade2-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:23:54.771057 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:54.771001    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qk9k\" (UniqueName: \"kubernetes.io/projected/2e656d6f-9d30-4600-b4ac-5fa8aa0cade2-kube-api-access-8qk9k\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:23:54.771057 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:54.771017    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e656d6f-9d30-4600-b4ac-5fa8aa0cade2-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:23:55.462551 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:55.462469    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wqs8h"
Nov 03 18:23:55.462551 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:55.462502    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wqs8h" event={"ID":"2e656d6f-9d30-4600-b4ac-5fa8aa0cade2","Type":"ContainerDied","Data":"dbbb2a01500e750c9cd0820bbd733efcc4d7ecf95ea0aaa825f0499ae79c225f"}
Nov 03 18:23:55.462762 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:23:55.462556    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbbb2a01500e750c9cd0820bbd733efcc4d7ecf95ea0aaa825f0499ae79c225f"
Nov 03 18:24:05.817348 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:05.817308    5315 scope.go:122] "RemoveContainer" containerID="568cabb51e7caf5bde5c2b68179ed7500cac1ec0f8a69a199de7625809a95cf2"
Nov 03 18:24:05.828117 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:05.828090    5315 scope.go:122] "RemoveContainer" containerID="8c74fbe22f327b2ee29d5d0c6ab5186e25693ac6516fe5ad69299c54e0abbbaf"
Nov 03 18:24:05.837346 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:05.837322    5315 scope.go:122] "RemoveContainer" containerID="d4d0a252baff91c18f15dd0a83dfdf8fcf5d66564446ab519cca916a8fb5b0a6"
Nov 03 18:24:05.845927 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:05.845909    5315 scope.go:122] "RemoveContainer" containerID="9af3796d4eaa197defae76ca6a355a21c41912f4120bd9008ab367439e851241"
Nov 03 18:24:05.853406 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:05.853389    5315 scope.go:122] "RemoveContainer" containerID="51b47b66423189c87ca8b0529db3ed767577870cf470a63753f78bf89b83c631"
Nov 03 18:24:05.860534 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:05.860513    5315 scope.go:122] "RemoveContainer" containerID="7917c425f9e3d4a5150b30ba950781197c304c3ca867c326ad2bef7f4633dcf9"
Nov 03 18:24:11.156801 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:11.156743    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-nd6sq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:24:11.157508 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:11.157474    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-nd6sq" podUID="cccc85e9-7f5c-4a60-a245-dacb32d91965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:24:11.181756 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:11.181713    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-7jkn9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:24:11.182067 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:11.182030    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7jkn9" podUID="0e37d948-2760-4441-8d44-722000f7e5b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:24:11.221750 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:11.221709    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-dzcvs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:24:11.222076 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:11.222042    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dzcvs" podUID="7e4a915b-dd46-4d77-b425-8c37440fdaad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:24:11.253777 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:11.253737    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-4jx7c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:24:11.254143 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:11.254101    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4jx7c" podUID="8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:24:19.413245 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:19.413196    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fr4v5"]
Nov 03 18:24:19.413833 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:19.413750    5315 cpu_manager.go:414] "containerMap: removing container" podUID="2e656d6f-9d30-4600-b4ac-5fa8aa0cade2" containerName="revoke-keys"
Nov 03 18:24:19.413833 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:19.413770    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e656d6f-9d30-4600-b4ac-5fa8aa0cade2" containerName="revoke-keys"
Nov 03 18:24:19.413954 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:19.413890    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e656d6f-9d30-4600-b4ac-5fa8aa0cade2" containerName="revoke-keys"
Nov 03 18:24:19.418989 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:19.418965    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fr4v5"
Nov 03 18:24:19.442923 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:19.442877    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7d79930-2bdb-46ed-8f43-2722e34f8948-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-fr4v5\" (UID: \"a7d79930-2bdb-46ed-8f43-2722e34f8948\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fr4v5"
Nov 03 18:24:19.443111 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:19.442938    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znx7z\" (UniqueName: \"kubernetes.io/projected/a7d79930-2bdb-46ed-8f43-2722e34f8948-kube-api-access-znx7z\") pod \"maas-api-revoke-sub-a4aef7d7-fr4v5\" (UID: \"a7d79930-2bdb-46ed-8f43-2722e34f8948\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fr4v5"
Nov 03 18:24:19.443190 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:19.443129    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7d79930-2bdb-46ed-8f43-2722e34f8948-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-fr4v5\" (UID: \"a7d79930-2bdb-46ed-8f43-2722e34f8948\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fr4v5"
Nov 03 18:24:19.467419 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:19.467385    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:24:19.513549 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:19.513517    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:24:19.513747 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:19.513586    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:24:19.513747 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:19.513666    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:24:19.515405 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:19.515386    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fr4v5"]
Nov 03 18:24:19.543927 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:19.543897    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7d79930-2bdb-46ed-8f43-2722e34f8948-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-fr4v5\" (UID: \"a7d79930-2bdb-46ed-8f43-2722e34f8948\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fr4v5"
Nov 03 18:24:19.544131 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:19.543938    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7d79930-2bdb-46ed-8f43-2722e34f8948-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-fr4v5\" (UID: \"a7d79930-2bdb-46ed-8f43-2722e34f8948\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fr4v5"
Nov 03 18:24:19.544131 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:19.543966    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znx7z\" (UniqueName: \"kubernetes.io/projected/a7d79930-2bdb-46ed-8f43-2722e34f8948-kube-api-access-znx7z\") pod \"maas-api-revoke-sub-a4aef7d7-fr4v5\" (UID: \"a7d79930-2bdb-46ed-8f43-2722e34f8948\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fr4v5"
Nov 03 18:24:19.544660 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:19.544634    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7d79930-2bdb-46ed-8f43-2722e34f8948-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-fr4v5\" (UID: \"a7d79930-2bdb-46ed-8f43-2722e34f8948\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fr4v5"
Nov 03 18:24:19.546358 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:19.546338    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7d79930-2bdb-46ed-8f43-2722e34f8948-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-fr4v5\" (UID: \"a7d79930-2bdb-46ed-8f43-2722e34f8948\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fr4v5"
Nov 03 18:24:19.601566 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:19.601531    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znx7z\" (UniqueName: \"kubernetes.io/projected/a7d79930-2bdb-46ed-8f43-2722e34f8948-kube-api-access-znx7z\") pod \"maas-api-revoke-sub-a4aef7d7-fr4v5\" (UID: \"a7d79930-2bdb-46ed-8f43-2722e34f8948\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fr4v5"
Nov 03 18:24:19.731480 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:19.731404    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fr4v5"
Nov 03 18:24:20.045131 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:20.045093    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2256a988-kv7fn"]
Nov 03 18:24:20.047840 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:20.047816    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2256a988-kv7fn"]
Nov 03 18:24:20.099251 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:20.099215    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fr4v5"]
Nov 03 18:24:20.142821 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:24:20.142779    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7d79930_2bdb_46ed_8f43_2722e34f8948.slice/crio-08970dc5152cffbaaa36559c6e7f8b704a06acf885e9e531b3c52eea28fca2d3 WatchSource:0}: Error finding container 08970dc5152cffbaaa36559c6e7f8b704a06acf885e9e531b3c52eea28fca2d3: Status 404 returned error can't find the container with id 08970dc5152cffbaaa36559c6e7f8b704a06acf885e9e531b3c52eea28fca2d3
Nov 03 18:24:20.598460 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:20.598357    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fr4v5" event={"ID":"a7d79930-2bdb-46ed-8f43-2722e34f8948","Type":"ContainerStarted","Data":"f63fa19daf59a1965e284bf4f0fa5d99d47e397bf22a81a23bd1faed4f856492"}
Nov 03 18:24:20.598460 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:20.598414    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fr4v5" event={"ID":"a7d79930-2bdb-46ed-8f43-2722e34f8948","Type":"ContainerStarted","Data":"08970dc5152cffbaaa36559c6e7f8b704a06acf885e9e531b3c52eea28fca2d3"}
Nov 03 18:24:20.707272 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:20.707215    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fr4v5" podStartSLOduration=1.7072009289999999 podStartE2EDuration="1.707200929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:24:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:24:20.706002766 +0000 UTC m=+2299.722200837" watchObservedRunningTime="2026-11-03 18:24:20.707200929 +0000 UTC m=+2299.723398998"
Nov 03 18:24:21.109080 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:21.109051    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06c7f90e-b35e-4c42-bc19-a98f8cd418fa" path="/var/lib/kubelet/pods/06c7f90e-b35e-4c42-bc19-a98f8cd418fa/volumes"
Nov 03 18:24:21.605399 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:21.605357    5315 generic.go:356] "Generic (PLEG): container finished" podID="a7d79930-2bdb-46ed-8f43-2722e34f8948" containerID="f63fa19daf59a1965e284bf4f0fa5d99d47e397bf22a81a23bd1faed4f856492" exitCode=0
Nov 03 18:24:21.605843 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:21.605444    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fr4v5" event={"ID":"a7d79930-2bdb-46ed-8f43-2722e34f8948","Type":"ContainerDied","Data":"f63fa19daf59a1965e284bf4f0fa5d99d47e397bf22a81a23bd1faed4f856492"}
Nov 03 18:24:22.739926 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:22.739901    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fr4v5"
Nov 03 18:24:22.775023 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:22.774996    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7d79930-2bdb-46ed-8f43-2722e34f8948-kube-api-access-znx7z\" (UniqueName: \"kubernetes.io/projected/a7d79930-2bdb-46ed-8f43-2722e34f8948-kube-api-access-znx7z\") pod \"a7d79930-2bdb-46ed-8f43-2722e34f8948\" (UID: \"a7d79930-2bdb-46ed-8f43-2722e34f8948\") "
Nov 03 18:24:22.775129 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:22.775061    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7d79930-2bdb-46ed-8f43-2722e34f8948-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7d79930-2bdb-46ed-8f43-2722e34f8948-maas-api-xks-ca\") pod \"a7d79930-2bdb-46ed-8f43-2722e34f8948\" (UID: \"a7d79930-2bdb-46ed-8f43-2722e34f8948\") "
Nov 03 18:24:22.775129 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:22.775089    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7d79930-2bdb-46ed-8f43-2722e34f8948-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7d79930-2bdb-46ed-8f43-2722e34f8948-maas-api-openshift-ca\") pod \"a7d79930-2bdb-46ed-8f43-2722e34f8948\" (UID: \"a7d79930-2bdb-46ed-8f43-2722e34f8948\") "
Nov 03 18:24:22.775480 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:22.775456    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7d79930-2bdb-46ed-8f43-2722e34f8948-maas-api-openshift-ca" pod "a7d79930-2bdb-46ed-8f43-2722e34f8948" (UID: "a7d79930-2bdb-46ed-8f43-2722e34f8948"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:24:22.777047 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:22.777024    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7d79930-2bdb-46ed-8f43-2722e34f8948-kube-api-access-znx7z" pod "a7d79930-2bdb-46ed-8f43-2722e34f8948" (UID: "a7d79930-2bdb-46ed-8f43-2722e34f8948"). InnerVolumeSpecName "kube-api-access-znx7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:24:22.777126 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:22.777097    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7d79930-2bdb-46ed-8f43-2722e34f8948-maas-api-xks-ca" pod "a7d79930-2bdb-46ed-8f43-2722e34f8948" (UID: "a7d79930-2bdb-46ed-8f43-2722e34f8948"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:24:22.875934 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:22.875860    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7d79930-2bdb-46ed-8f43-2722e34f8948-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:24:22.875934 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:22.875898    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7d79930-2bdb-46ed-8f43-2722e34f8948-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:24:22.875934 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:22.875916    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znx7z\" (UniqueName: \"kubernetes.io/projected/a7d79930-2bdb-46ed-8f43-2722e34f8948-kube-api-access-znx7z\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:24:23.616011 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:23.615976    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fr4v5"
Nov 03 18:24:23.616011 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:23.615999    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fr4v5" event={"ID":"a7d79930-2bdb-46ed-8f43-2722e34f8948","Type":"ContainerDied","Data":"08970dc5152cffbaaa36559c6e7f8b704a06acf885e9e531b3c52eea28fca2d3"}
Nov 03 18:24:23.616305 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:23.616045    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08970dc5152cffbaaa36559c6e7f8b704a06acf885e9e531b3c52eea28fca2d3"
Nov 03 18:24:24.576808 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:24:24.576766    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:24:41.160771 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:41.160723    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-nd6sq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:24:41.161457 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:41.161424    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-nd6sq" podUID="cccc85e9-7f5c-4a60-a245-dacb32d91965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:24:41.183636 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:41.181683    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-7jkn9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:24:41.183636 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:41.181748    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7jkn9" podUID="0e37d948-2760-4441-8d44-722000f7e5b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:24:41.225633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:41.222765    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-dzcvs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:24:41.225633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:41.222849    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dzcvs" podUID="7e4a915b-dd46-4d77-b425-8c37440fdaad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:24:41.255633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:41.254734    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-4jx7c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:24:41.255633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:41.254805    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4jx7c" podUID="8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:24:46.046886 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:46.046842    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa531960-p6vw5"]
Nov 03 18:24:46.093027 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:46.092988    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa531960-p6vw5"]
Nov 03 18:24:47.102745 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:47.102708    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f7a62e2-84ad-4c09-980e-d69d77d69531" path="/var/lib/kubelet/pods/9f7a62e2-84ad-4c09-980e-d69d77d69531/volumes"
Nov 03 18:24:50.243430 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:50.243395    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vrlz9"]
Nov 03 18:24:50.243845 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:50.243799    5315 cpu_manager.go:414] "containerMap: removing container" podUID="a7d79930-2bdb-46ed-8f43-2722e34f8948" containerName="revoke-keys"
Nov 03 18:24:50.243845 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:50.243812    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7d79930-2bdb-46ed-8f43-2722e34f8948" containerName="revoke-keys"
Nov 03 18:24:50.243943 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:50.243874    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7d79930-2bdb-46ed-8f43-2722e34f8948" containerName="revoke-keys"
Nov 03 18:24:50.247374 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:50.247353    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vrlz9"
Nov 03 18:24:50.295536 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:50.295503    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dtz5\" (UniqueName: \"kubernetes.io/projected/c0196849-40f8-4d47-a66d-673b84140384-kube-api-access-9dtz5\") pod \"maas-api-revoke-sub-79805ca9-vrlz9\" (UID: \"c0196849-40f8-4d47-a66d-673b84140384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vrlz9"
Nov 03 18:24:50.295751 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:50.295556    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0196849-40f8-4d47-a66d-673b84140384-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-vrlz9\" (UID: \"c0196849-40f8-4d47-a66d-673b84140384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vrlz9"
Nov 03 18:24:50.295751 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:50.295719    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0196849-40f8-4d47-a66d-673b84140384-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-vrlz9\" (UID: \"c0196849-40f8-4d47-a66d-673b84140384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vrlz9"
Nov 03 18:24:50.296145 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:50.296118    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:24:50.296145 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:50.296127    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:24:50.342255 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:50.342216    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:24:50.342424 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:50.342406    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:24:50.344050 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:50.344031    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vrlz9"]
Nov 03 18:24:50.396453 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:50.396421    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0196849-40f8-4d47-a66d-673b84140384-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-vrlz9\" (UID: \"c0196849-40f8-4d47-a66d-673b84140384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vrlz9"
Nov 03 18:24:50.396657 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:50.396489    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dtz5\" (UniqueName: \"kubernetes.io/projected/c0196849-40f8-4d47-a66d-673b84140384-kube-api-access-9dtz5\") pod \"maas-api-revoke-sub-79805ca9-vrlz9\" (UID: \"c0196849-40f8-4d47-a66d-673b84140384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vrlz9"
Nov 03 18:24:50.396657 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:50.396532    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0196849-40f8-4d47-a66d-673b84140384-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-vrlz9\" (UID: \"c0196849-40f8-4d47-a66d-673b84140384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vrlz9"
Nov 03 18:24:50.397109 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:50.397087    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0196849-40f8-4d47-a66d-673b84140384-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-vrlz9\" (UID: \"c0196849-40f8-4d47-a66d-673b84140384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vrlz9"
Nov 03 18:24:50.398989 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:50.398931    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0196849-40f8-4d47-a66d-673b84140384-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-vrlz9\" (UID: \"c0196849-40f8-4d47-a66d-673b84140384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vrlz9"
Nov 03 18:24:50.454300 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:50.454267    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dtz5\" (UniqueName: \"kubernetes.io/projected/c0196849-40f8-4d47-a66d-673b84140384-kube-api-access-9dtz5\") pod \"maas-api-revoke-sub-79805ca9-vrlz9\" (UID: \"c0196849-40f8-4d47-a66d-673b84140384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vrlz9"
Nov 03 18:24:50.559764 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:50.559730    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vrlz9"
Nov 03 18:24:50.928845 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:50.928763    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vrlz9"]
Nov 03 18:24:50.935391 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:24:50.935355    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0196849_40f8_4d47_a66d_673b84140384.slice/crio-5448a83505bd8082c5ee5808c88e41684c733c7bdc5cdccab7828ed31b045e77 WatchSource:0}: Error finding container 5448a83505bd8082c5ee5808c88e41684c733c7bdc5cdccab7828ed31b045e77: Status 404 returned error can't find the container with id 5448a83505bd8082c5ee5808c88e41684c733c7bdc5cdccab7828ed31b045e77
Nov 03 18:24:51.786316 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:51.786278    5315 generic.go:356] "Generic (PLEG): container finished" podID="c0196849-40f8-4d47-a66d-673b84140384" containerID="15b0b35ee20019a7a6c39fe92da43ab28892948be81c33508b7226e01fb51c32" exitCode=0
Nov 03 18:24:51.786729 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:51.786365    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vrlz9" event={"ID":"c0196849-40f8-4d47-a66d-673b84140384","Type":"ContainerDied","Data":"15b0b35ee20019a7a6c39fe92da43ab28892948be81c33508b7226e01fb51c32"}
Nov 03 18:24:51.786729 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:51.786420    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vrlz9" event={"ID":"c0196849-40f8-4d47-a66d-673b84140384","Type":"ContainerStarted","Data":"5448a83505bd8082c5ee5808c88e41684c733c7bdc5cdccab7828ed31b045e77"}
Nov 03 18:24:52.927313 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:52.927288    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vrlz9"
Nov 03 18:24:53.015811 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:53.015773    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0196849-40f8-4d47-a66d-673b84140384-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0196849-40f8-4d47-a66d-673b84140384-maas-api-openshift-ca\") pod \"c0196849-40f8-4d47-a66d-673b84140384\" (UID: \"c0196849-40f8-4d47-a66d-673b84140384\") "
Nov 03 18:24:53.016003 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:53.015844    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0196849-40f8-4d47-a66d-673b84140384-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0196849-40f8-4d47-a66d-673b84140384-maas-api-xks-ca\") pod \"c0196849-40f8-4d47-a66d-673b84140384\" (UID: \"c0196849-40f8-4d47-a66d-673b84140384\") "
Nov 03 18:24:53.016003 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:53.015887    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0196849-40f8-4d47-a66d-673b84140384-kube-api-access-9dtz5\" (UniqueName: \"kubernetes.io/projected/c0196849-40f8-4d47-a66d-673b84140384-kube-api-access-9dtz5\") pod \"c0196849-40f8-4d47-a66d-673b84140384\" (UID: \"c0196849-40f8-4d47-a66d-673b84140384\") "
Nov 03 18:24:53.016247 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:53.016216    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0196849-40f8-4d47-a66d-673b84140384-maas-api-openshift-ca" pod "c0196849-40f8-4d47-a66d-673b84140384" (UID: "c0196849-40f8-4d47-a66d-673b84140384"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:24:53.018011 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:53.017980    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0196849-40f8-4d47-a66d-673b84140384-kube-api-access-9dtz5" pod "c0196849-40f8-4d47-a66d-673b84140384" (UID: "c0196849-40f8-4d47-a66d-673b84140384"). InnerVolumeSpecName "kube-api-access-9dtz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:24:53.018136 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:53.018039    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0196849-40f8-4d47-a66d-673b84140384-maas-api-xks-ca" pod "c0196849-40f8-4d47-a66d-673b84140384" (UID: "c0196849-40f8-4d47-a66d-673b84140384"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:24:53.117376 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:53.117346    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0196849-40f8-4d47-a66d-673b84140384-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:24:53.117376 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:53.117373    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0196849-40f8-4d47-a66d-673b84140384-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:24:53.117562 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:53.117385    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dtz5\" (UniqueName: \"kubernetes.io/projected/c0196849-40f8-4d47-a66d-673b84140384-kube-api-access-9dtz5\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:24:53.796968 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:53.796929    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vrlz9" event={"ID":"c0196849-40f8-4d47-a66d-673b84140384","Type":"ContainerDied","Data":"5448a83505bd8082c5ee5808c88e41684c733c7bdc5cdccab7828ed31b045e77"}
Nov 03 18:24:53.796968 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:53.796963    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vrlz9"
Nov 03 18:24:53.797274 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:24:53.796994    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5448a83505bd8082c5ee5808c88e41684c733c7bdc5cdccab7828ed31b045e77"
Nov 03 18:24:54.577112 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:24:54.577075    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:25:11.084331 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:11.084292    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cslvm"]
Nov 03 18:25:11.084770 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:11.084687    5315 cpu_manager.go:414] "containerMap: removing container" podUID="c0196849-40f8-4d47-a66d-673b84140384" containerName="revoke-keys"
Nov 03 18:25:11.084770 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:11.084701    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0196849-40f8-4d47-a66d-673b84140384" containerName="revoke-keys"
Nov 03 18:25:11.084846 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:11.084800    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0196849-40f8-4d47-a66d-673b84140384" containerName="revoke-keys"
Nov 03 18:25:11.088236 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:11.088205    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cslvm"
Nov 03 18:25:11.137021 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:11.136980    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:25:11.137193 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:11.137019    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:25:11.141422 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:11.141397    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8673571d-193d-4120-b63a-e517f961d06f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-cslvm\" (UID: \"8673571d-193d-4120-b63a-e517f961d06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cslvm"
Nov 03 18:25:11.141557 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:11.141536    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8673571d-193d-4120-b63a-e517f961d06f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-cslvm\" (UID: \"8673571d-193d-4120-b63a-e517f961d06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cslvm"
Nov 03 18:25:11.141649 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:11.141621    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmpnn\" (UniqueName: \"kubernetes.io/projected/8673571d-193d-4120-b63a-e517f961d06f-kube-api-access-qmpnn\") pod \"maas-api-revoke-sub-0b7e8edb-cslvm\" (UID: \"8673571d-193d-4120-b63a-e517f961d06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cslvm"
Nov 03 18:25:11.183201 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:11.183163    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:25:11.183391 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:11.183291    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:25:11.185114 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:11.185093    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cslvm"]
Nov 03 18:25:11.242766 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:11.242724    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8673571d-193d-4120-b63a-e517f961d06f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-cslvm\" (UID: \"8673571d-193d-4120-b63a-e517f961d06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cslvm"
Nov 03 18:25:11.242972 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:11.242846    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8673571d-193d-4120-b63a-e517f961d06f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-cslvm\" (UID: \"8673571d-193d-4120-b63a-e517f961d06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cslvm"
Nov 03 18:25:11.242972 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:11.242885    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmpnn\" (UniqueName: \"kubernetes.io/projected/8673571d-193d-4120-b63a-e517f961d06f-kube-api-access-qmpnn\") pod \"maas-api-revoke-sub-0b7e8edb-cslvm\" (UID: \"8673571d-193d-4120-b63a-e517f961d06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cslvm"
Nov 03 18:25:11.243518 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:11.243495    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8673571d-193d-4120-b63a-e517f961d06f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-cslvm\" (UID: \"8673571d-193d-4120-b63a-e517f961d06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cslvm"
Nov 03 18:25:11.245209 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:11.245186    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8673571d-193d-4120-b63a-e517f961d06f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-cslvm\" (UID: \"8673571d-193d-4120-b63a-e517f961d06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cslvm"
Nov 03 18:25:11.300356 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:11.300317    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmpnn\" (UniqueName: \"kubernetes.io/projected/8673571d-193d-4120-b63a-e517f961d06f-kube-api-access-qmpnn\") pod \"maas-api-revoke-sub-0b7e8edb-cslvm\" (UID: \"8673571d-193d-4120-b63a-e517f961d06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cslvm"
Nov 03 18:25:11.401771 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:11.401675    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cslvm"
Nov 03 18:25:11.769335 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:11.769231    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cslvm"]
Nov 03 18:25:11.776279 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:25:11.776249    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8673571d_193d_4120_b63a_e517f961d06f.slice/crio-5d0bfe40355f3de40bb115df97039b0d34ac7edc85164467a235e5342e44b939 WatchSource:0}: Error finding container 5d0bfe40355f3de40bb115df97039b0d34ac7edc85164467a235e5342e44b939: Status 404 returned error can't find the container with id 5d0bfe40355f3de40bb115df97039b0d34ac7edc85164467a235e5342e44b939
Nov 03 18:25:11.941187 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:11.941113    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cslvm" event={"ID":"8673571d-193d-4120-b63a-e517f961d06f","Type":"ContainerStarted","Data":"5d0bfe40355f3de40bb115df97039b0d34ac7edc85164467a235e5342e44b939"}
Nov 03 18:25:12.947656 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:12.947547    5315 generic.go:356] "Generic (PLEG): container finished" podID="8673571d-193d-4120-b63a-e517f961d06f" containerID="e82eeb9313d7715ea9b1af20cf2694dc31d94306ec49637b7542bbd6a7b0ad23" exitCode=0
Nov 03 18:25:12.947656 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:12.947628    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cslvm" event={"ID":"8673571d-193d-4120-b63a-e517f961d06f","Type":"ContainerDied","Data":"e82eeb9313d7715ea9b1af20cf2694dc31d94306ec49637b7542bbd6a7b0ad23"}
Nov 03 18:25:14.095287 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:14.095259    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cslvm"
Nov 03 18:25:14.171049 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:14.171012    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8673571d-193d-4120-b63a-e517f961d06f-kube-api-access-qmpnn\" (UniqueName: \"kubernetes.io/projected/8673571d-193d-4120-b63a-e517f961d06f-kube-api-access-qmpnn\") pod \"8673571d-193d-4120-b63a-e517f961d06f\" (UID: \"8673571d-193d-4120-b63a-e517f961d06f\") "
Nov 03 18:25:14.171211 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:14.171180    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8673571d-193d-4120-b63a-e517f961d06f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8673571d-193d-4120-b63a-e517f961d06f-maas-api-xks-ca\") pod \"8673571d-193d-4120-b63a-e517f961d06f\" (UID: \"8673571d-193d-4120-b63a-e517f961d06f\") "
Nov 03 18:25:14.171283 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:14.171227    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8673571d-193d-4120-b63a-e517f961d06f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8673571d-193d-4120-b63a-e517f961d06f-maas-api-openshift-ca\") pod \"8673571d-193d-4120-b63a-e517f961d06f\" (UID: \"8673571d-193d-4120-b63a-e517f961d06f\") "
Nov 03 18:25:14.171605 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:14.171571    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8673571d-193d-4120-b63a-e517f961d06f-maas-api-openshift-ca" pod "8673571d-193d-4120-b63a-e517f961d06f" (UID: "8673571d-193d-4120-b63a-e517f961d06f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:25:14.173160 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:14.173136    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8673571d-193d-4120-b63a-e517f961d06f-maas-api-xks-ca" pod "8673571d-193d-4120-b63a-e517f961d06f" (UID: "8673571d-193d-4120-b63a-e517f961d06f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:25:14.173237 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:14.173207    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8673571d-193d-4120-b63a-e517f961d06f-kube-api-access-qmpnn" pod "8673571d-193d-4120-b63a-e517f961d06f" (UID: "8673571d-193d-4120-b63a-e517f961d06f"). InnerVolumeSpecName "kube-api-access-qmpnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:25:14.272967 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:14.272871    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8673571d-193d-4120-b63a-e517f961d06f-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:25:14.272967 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:14.272915    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8673571d-193d-4120-b63a-e517f961d06f-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:25:14.272967 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:14.272927    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmpnn\" (UniqueName: \"kubernetes.io/projected/8673571d-193d-4120-b63a-e517f961d06f-kube-api-access-qmpnn\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:25:14.958825 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:14.958788    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cslvm" event={"ID":"8673571d-193d-4120-b63a-e517f961d06f","Type":"ContainerDied","Data":"5d0bfe40355f3de40bb115df97039b0d34ac7edc85164467a235e5342e44b939"}
Nov 03 18:25:14.959045 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:14.958835    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d0bfe40355f3de40bb115df97039b0d34ac7edc85164467a235e5342e44b939"
Nov 03 18:25:14.959045 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:14.958837    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cslvm"
Nov 03 18:25:22.496241 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:22.496201    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9c4pv"]
Nov 03 18:25:22.496711 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:22.496551    5315 cpu_manager.go:414] "containerMap: removing container" podUID="8673571d-193d-4120-b63a-e517f961d06f" containerName="revoke-keys"
Nov 03 18:25:22.496711 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:22.496562    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8673571d-193d-4120-b63a-e517f961d06f" containerName="revoke-keys"
Nov 03 18:25:22.496711 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:22.496645    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="8673571d-193d-4120-b63a-e517f961d06f" containerName="revoke-keys"
Nov 03 18:25:22.499761 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:22.499737    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9c4pv"
Nov 03 18:25:22.548431 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:22.548402    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:25:22.594564 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:22.594522    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:25:22.594564 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:22.594530    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:25:22.594839 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:22.594581    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:25:22.596428 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:22.596401    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9c4pv"]
Nov 03 18:25:22.657047 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:22.657012    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41c69217-cb53-4025-b0af-33f15f52caf2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9c4pv\" (UID: \"41c69217-cb53-4025-b0af-33f15f52caf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9c4pv"
Nov 03 18:25:22.657227 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:22.657058    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh78b\" (UniqueName: \"kubernetes.io/projected/41c69217-cb53-4025-b0af-33f15f52caf2-kube-api-access-zh78b\") pod \"maas-api-revoke-sub-3d4156f1-9c4pv\" (UID: \"41c69217-cb53-4025-b0af-33f15f52caf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9c4pv"
Nov 03 18:25:22.657227 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:22.657162    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41c69217-cb53-4025-b0af-33f15f52caf2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9c4pv\" (UID: \"41c69217-cb53-4025-b0af-33f15f52caf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9c4pv"
Nov 03 18:25:22.758291 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:22.758191    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41c69217-cb53-4025-b0af-33f15f52caf2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9c4pv\" (UID: \"41c69217-cb53-4025-b0af-33f15f52caf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9c4pv"
Nov 03 18:25:22.758291 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:22.758250    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh78b\" (UniqueName: \"kubernetes.io/projected/41c69217-cb53-4025-b0af-33f15f52caf2-kube-api-access-zh78b\") pod \"maas-api-revoke-sub-3d4156f1-9c4pv\" (UID: \"41c69217-cb53-4025-b0af-33f15f52caf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9c4pv"
Nov 03 18:25:22.758548 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:22.758298    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41c69217-cb53-4025-b0af-33f15f52caf2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9c4pv\" (UID: \"41c69217-cb53-4025-b0af-33f15f52caf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9c4pv"
Nov 03 18:25:22.758963 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:22.758937    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41c69217-cb53-4025-b0af-33f15f52caf2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9c4pv\" (UID: \"41c69217-cb53-4025-b0af-33f15f52caf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9c4pv"
Nov 03 18:25:22.760696 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:22.760673    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41c69217-cb53-4025-b0af-33f15f52caf2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9c4pv\" (UID: \"41c69217-cb53-4025-b0af-33f15f52caf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9c4pv"
Nov 03 18:25:22.815248 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:22.815215    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh78b\" (UniqueName: \"kubernetes.io/projected/41c69217-cb53-4025-b0af-33f15f52caf2-kube-api-access-zh78b\") pod \"maas-api-revoke-sub-3d4156f1-9c4pv\" (UID: \"41c69217-cb53-4025-b0af-33f15f52caf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9c4pv"
Nov 03 18:25:23.113855 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:23.113818    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9c4pv"
Nov 03 18:25:23.482001 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:23.481896    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9c4pv"]
Nov 03 18:25:23.488890 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:25:23.488858    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41c69217_cb53_4025_b0af_33f15f52caf2.slice/crio-ac48cea208621e47192f274791c09415e4b63bda0e6c802ac0502234ece2c944 WatchSource:0}: Error finding container ac48cea208621e47192f274791c09415e4b63bda0e6c802ac0502234ece2c944: Status 404 returned error can't find the container with id ac48cea208621e47192f274791c09415e4b63bda0e6c802ac0502234ece2c944
Nov 03 18:25:24.009578 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:24.009533    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9c4pv" event={"ID":"41c69217-cb53-4025-b0af-33f15f52caf2","Type":"ContainerStarted","Data":"1ddbc662389ac4fc92fe01a6a96c9db09e50e8a83d29142f9614c38b7956e452"}
Nov 03 18:25:24.010003 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:24.009646    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9c4pv" event={"ID":"41c69217-cb53-4025-b0af-33f15f52caf2","Type":"ContainerStarted","Data":"ac48cea208621e47192f274791c09415e4b63bda0e6c802ac0502234ece2c944"}
Nov 03 18:25:24.123516 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:24.123461    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9c4pv" podStartSLOduration=2.123446071 podStartE2EDuration="2.123446071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:25:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:25:24.122162718 +0000 UTC m=+2363.138360788" watchObservedRunningTime="2026-11-03 18:25:24.123446071 +0000 UTC m=+2363.139644140"
Nov 03 18:25:24.715635 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:25:24.714799    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:25:25.034959 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:25.034919    5315 generic.go:356] "Generic (PLEG): container finished" podID="41c69217-cb53-4025-b0af-33f15f52caf2" containerID="1ddbc662389ac4fc92fe01a6a96c9db09e50e8a83d29142f9614c38b7956e452" exitCode=0
Nov 03 18:25:25.035560 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:25.035533    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9c4pv" event={"ID":"41c69217-cb53-4025-b0af-33f15f52caf2","Type":"ContainerDied","Data":"1ddbc662389ac4fc92fe01a6a96c9db09e50e8a83d29142f9614c38b7956e452"}
Nov 03 18:25:26.147635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:26.146723    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-nd6sq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:25:26.147635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:26.146798    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-nd6sq" podUID="cccc85e9-7f5c-4a60-a245-dacb32d91965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:25:26.191756 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:26.191715    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-7jkn9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:25:26.192046 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:26.192015    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7jkn9" podUID="0e37d948-2760-4441-8d44-722000f7e5b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:25:26.223631 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:26.222741    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-dzcvs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:25:26.223631 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:26.222821    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dzcvs" podUID="7e4a915b-dd46-4d77-b425-8c37440fdaad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:25:26.262713 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:26.258737    5315 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-4jx7c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:25:26.262713 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:26.258803    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-4jx7c" podUID="8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:25:26.442990 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:26.442961    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9c4pv"
Nov 03 18:25:26.530448 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:26.530406    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41c69217-cb53-4025-b0af-33f15f52caf2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41c69217-cb53-4025-b0af-33f15f52caf2-maas-api-xks-ca\") pod \"41c69217-cb53-4025-b0af-33f15f52caf2\" (UID: \"41c69217-cb53-4025-b0af-33f15f52caf2\") "
Nov 03 18:25:26.530673 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:26.530477    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41c69217-cb53-4025-b0af-33f15f52caf2-kube-api-access-zh78b\" (UniqueName: \"kubernetes.io/projected/41c69217-cb53-4025-b0af-33f15f52caf2-kube-api-access-zh78b\") pod \"41c69217-cb53-4025-b0af-33f15f52caf2\" (UID: \"41c69217-cb53-4025-b0af-33f15f52caf2\") "
Nov 03 18:25:26.530673 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:26.530585    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41c69217-cb53-4025-b0af-33f15f52caf2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41c69217-cb53-4025-b0af-33f15f52caf2-maas-api-openshift-ca\") pod \"41c69217-cb53-4025-b0af-33f15f52caf2\" (UID: \"41c69217-cb53-4025-b0af-33f15f52caf2\") "
Nov 03 18:25:26.530940 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:26.530915    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41c69217-cb53-4025-b0af-33f15f52caf2-maas-api-openshift-ca" pod "41c69217-cb53-4025-b0af-33f15f52caf2" (UID: "41c69217-cb53-4025-b0af-33f15f52caf2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:25:26.532650 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:26.532624    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41c69217-cb53-4025-b0af-33f15f52caf2-kube-api-access-zh78b" pod "41c69217-cb53-4025-b0af-33f15f52caf2" (UID: "41c69217-cb53-4025-b0af-33f15f52caf2"). InnerVolumeSpecName "kube-api-access-zh78b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:25:26.532750 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:26.532646    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41c69217-cb53-4025-b0af-33f15f52caf2-maas-api-xks-ca" pod "41c69217-cb53-4025-b0af-33f15f52caf2" (UID: "41c69217-cb53-4025-b0af-33f15f52caf2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:25:26.632091 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:26.632043    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41c69217-cb53-4025-b0af-33f15f52caf2-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:25:26.632091 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:26.632083    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh78b\" (UniqueName: \"kubernetes.io/projected/41c69217-cb53-4025-b0af-33f15f52caf2-kube-api-access-zh78b\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:25:26.632091 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:26.632099    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41c69217-cb53-4025-b0af-33f15f52caf2-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:25:27.059471 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:27.059429    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9c4pv"
Nov 03 18:25:27.059471 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:27.059462    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9c4pv" event={"ID":"41c69217-cb53-4025-b0af-33f15f52caf2","Type":"ContainerDied","Data":"ac48cea208621e47192f274791c09415e4b63bda0e6c802ac0502234ece2c944"}
Nov 03 18:25:27.059736 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:27.059510    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac48cea208621e47192f274791c09415e4b63bda0e6c802ac0502234ece2c944"
Nov 03 18:25:43.048244 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:43.048197    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-5fjng"]
Nov 03 18:25:43.094106 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:43.094069    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-5fjng"]
Nov 03 18:25:43.102895 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:43.102863    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88de2dfb-9252-405d-b590-abf68ec6dbe9" path="/var/lib/kubelet/pods/88de2dfb-9252-405d-b590-abf68ec6dbe9/volumes"
Nov 03 18:25:43.769655 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:43.769603    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9c4pv"]
Nov 03 18:25:43.815760 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:43.815723    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9c4pv"]
Nov 03 18:25:43.890904 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:43.890866    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4x9x9"]
Nov 03 18:25:43.891258 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:43.891245    5315 cpu_manager.go:414] "containerMap: removing container" podUID="41c69217-cb53-4025-b0af-33f15f52caf2" containerName="revoke-keys"
Nov 03 18:25:43.891307 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:43.891260    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41c69217-cb53-4025-b0af-33f15f52caf2" containerName="revoke-keys"
Nov 03 18:25:43.891344 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:43.891328    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="41c69217-cb53-4025-b0af-33f15f52caf2" containerName="revoke-keys"
Nov 03 18:25:43.894668 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:43.894641    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4x9x9"
Nov 03 18:25:43.943207 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:43.943171    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:25:43.943207 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:43.943224    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:25:43.943447 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:43.943327    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:25:43.943447 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:43.943343    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:25:43.954562 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:43.954538    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4x9x9"]
Nov 03 18:25:44.029733 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:44.029704    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f7eb77f-2359-46f0-aa39-67dbff914c81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4x9x9\" (UID: \"8f7eb77f-2359-46f0-aa39-67dbff914c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4x9x9"
Nov 03 18:25:44.029906 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:44.029769    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sltdx\" (UniqueName: \"kubernetes.io/projected/8f7eb77f-2359-46f0-aa39-67dbff914c81-kube-api-access-sltdx\") pod \"maas-api-revoke-sub-3d4156f1-4x9x9\" (UID: \"8f7eb77f-2359-46f0-aa39-67dbff914c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4x9x9"
Nov 03 18:25:44.029964 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:44.029899    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f7eb77f-2359-46f0-aa39-67dbff914c81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4x9x9\" (UID: \"8f7eb77f-2359-46f0-aa39-67dbff914c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4x9x9"
Nov 03 18:25:44.130394 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:44.130358    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sltdx\" (UniqueName: \"kubernetes.io/projected/8f7eb77f-2359-46f0-aa39-67dbff914c81-kube-api-access-sltdx\") pod \"maas-api-revoke-sub-3d4156f1-4x9x9\" (UID: \"8f7eb77f-2359-46f0-aa39-67dbff914c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4x9x9"
Nov 03 18:25:44.130868 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:44.130409    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f7eb77f-2359-46f0-aa39-67dbff914c81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4x9x9\" (UID: \"8f7eb77f-2359-46f0-aa39-67dbff914c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4x9x9"
Nov 03 18:25:44.130868 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:44.130458    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f7eb77f-2359-46f0-aa39-67dbff914c81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4x9x9\" (UID: \"8f7eb77f-2359-46f0-aa39-67dbff914c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4x9x9"
Nov 03 18:25:44.131059 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:44.131035    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f7eb77f-2359-46f0-aa39-67dbff914c81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4x9x9\" (UID: \"8f7eb77f-2359-46f0-aa39-67dbff914c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4x9x9"
Nov 03 18:25:44.132908 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:44.132883    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f7eb77f-2359-46f0-aa39-67dbff914c81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4x9x9\" (UID: \"8f7eb77f-2359-46f0-aa39-67dbff914c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4x9x9"
Nov 03 18:25:44.187436 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:44.187400    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sltdx\" (UniqueName: \"kubernetes.io/projected/8f7eb77f-2359-46f0-aa39-67dbff914c81-kube-api-access-sltdx\") pod \"maas-api-revoke-sub-3d4156f1-4x9x9\" (UID: \"8f7eb77f-2359-46f0-aa39-67dbff914c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4x9x9"
Nov 03 18:25:44.207400 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:44.207365    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4x9x9"
Nov 03 18:25:44.575917 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:44.575880    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4x9x9"]
Nov 03 18:25:44.582835 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:25:44.582798    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f7eb77f_2359_46f0_aa39_67dbff914c81.slice/crio-2e8588e1509b150dd818f761dd84e488b8f5d37e26743c9c0a9261f1f9140b0e WatchSource:0}: Error finding container 2e8588e1509b150dd818f761dd84e488b8f5d37e26743c9c0a9261f1f9140b0e: Status 404 returned error can't find the container with id 2e8588e1509b150dd818f761dd84e488b8f5d37e26743c9c0a9261f1f9140b0e
Nov 03 18:25:45.103463 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:45.103385    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41c69217-cb53-4025-b0af-33f15f52caf2" path="/var/lib/kubelet/pods/41c69217-cb53-4025-b0af-33f15f52caf2/volumes"
Nov 03 18:25:45.171954 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:45.171881    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4x9x9" event={"ID":"8f7eb77f-2359-46f0-aa39-67dbff914c81","Type":"ContainerStarted","Data":"7643a7e8fa25648c0f70a7440b61b390220e19b4c2d73252cad3dd77965d754a"}
Nov 03 18:25:45.171954 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:45.171948    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4x9x9" event={"ID":"8f7eb77f-2359-46f0-aa39-67dbff914c81","Type":"ContainerStarted","Data":"2e8588e1509b150dd818f761dd84e488b8f5d37e26743c9c0a9261f1f9140b0e"}
Nov 03 18:25:45.280498 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:45.280437    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4x9x9" podStartSLOduration=2.280422537 podStartE2EDuration="2.280422537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:25:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:25:45.279374993 +0000 UTC m=+2384.295573064" watchObservedRunningTime="2026-11-03 18:25:45.280422537 +0000 UTC m=+2384.296620605"
Nov 03 18:25:46.177685 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:46.177649    5315 generic.go:356] "Generic (PLEG): container finished" podID="8f7eb77f-2359-46f0-aa39-67dbff914c81" containerID="7643a7e8fa25648c0f70a7440b61b390220e19b4c2d73252cad3dd77965d754a" exitCode=0
Nov 03 18:25:46.178096 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:46.177735    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4x9x9" event={"ID":"8f7eb77f-2359-46f0-aa39-67dbff914c81","Type":"ContainerDied","Data":"7643a7e8fa25648c0f70a7440b61b390220e19b4c2d73252cad3dd77965d754a"}
Nov 03 18:25:47.327379 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:47.327354    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4x9x9"
Nov 03 18:25:47.464861 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:47.464772    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f7eb77f-2359-46f0-aa39-67dbff914c81-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f7eb77f-2359-46f0-aa39-67dbff914c81-maas-api-xks-ca\") pod \"8f7eb77f-2359-46f0-aa39-67dbff914c81\" (UID: \"8f7eb77f-2359-46f0-aa39-67dbff914c81\") "
Nov 03 18:25:47.464861 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:47.464812    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f7eb77f-2359-46f0-aa39-67dbff914c81-kube-api-access-sltdx\" (UniqueName: \"kubernetes.io/projected/8f7eb77f-2359-46f0-aa39-67dbff914c81-kube-api-access-sltdx\") pod \"8f7eb77f-2359-46f0-aa39-67dbff914c81\" (UID: \"8f7eb77f-2359-46f0-aa39-67dbff914c81\") "
Nov 03 18:25:47.464861 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:47.464835    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f7eb77f-2359-46f0-aa39-67dbff914c81-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f7eb77f-2359-46f0-aa39-67dbff914c81-maas-api-openshift-ca\") pod \"8f7eb77f-2359-46f0-aa39-67dbff914c81\" (UID: \"8f7eb77f-2359-46f0-aa39-67dbff914c81\") "
Nov 03 18:25:47.465269 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:47.465244    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f7eb77f-2359-46f0-aa39-67dbff914c81-maas-api-openshift-ca" pod "8f7eb77f-2359-46f0-aa39-67dbff914c81" (UID: "8f7eb77f-2359-46f0-aa39-67dbff914c81"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:25:47.466984 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:47.466952    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f7eb77f-2359-46f0-aa39-67dbff914c81-maas-api-xks-ca" pod "8f7eb77f-2359-46f0-aa39-67dbff914c81" (UID: "8f7eb77f-2359-46f0-aa39-67dbff914c81"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:25:47.467090 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:47.467071    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f7eb77f-2359-46f0-aa39-67dbff914c81-kube-api-access-sltdx" pod "8f7eb77f-2359-46f0-aa39-67dbff914c81" (UID: "8f7eb77f-2359-46f0-aa39-67dbff914c81"). InnerVolumeSpecName "kube-api-access-sltdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:25:47.566407 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:47.566368    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f7eb77f-2359-46f0-aa39-67dbff914c81-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:25:47.566407 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:47.566401    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sltdx\" (UniqueName: \"kubernetes.io/projected/8f7eb77f-2359-46f0-aa39-67dbff914c81-kube-api-access-sltdx\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:25:47.566407 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:47.566411    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f7eb77f-2359-46f0-aa39-67dbff914c81-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:25:48.188447 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:48.188413    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4x9x9"
Nov 03 18:25:48.188659 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:48.188423    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4x9x9" event={"ID":"8f7eb77f-2359-46f0-aa39-67dbff914c81","Type":"ContainerDied","Data":"2e8588e1509b150dd818f761dd84e488b8f5d37e26743c9c0a9261f1f9140b0e"}
Nov 03 18:25:48.188659 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:25:48.188538    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e8588e1509b150dd818f761dd84e488b8f5d37e26743c9c0a9261f1f9140b0e"
Nov 03 18:25:54.576532 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:25:54.576487    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:26:01.654751 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:01.654718    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-97p65_c29d200a-c917-46e5-9cf5-21f4d952e470/ovn-acl-logging/0.log"
Nov 03 18:26:01.702802 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:01.702776    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-97p65_c29d200a-c917-46e5-9cf5-21f4d952e470/ovn-acl-logging/0.log"
Nov 03 18:26:05.908988 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:05.908950    5315 scope.go:122] "RemoveContainer" containerID="3d5c97b2587474fc715cf28ce9fe4efa2d96afe5c41619726466df69c752e137"
Nov 03 18:26:05.918782 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:05.918754    5315 scope.go:122] "RemoveContainer" containerID="a4f2216c44649b4a11d31532fab0679df82f7bfde1ccf513b3f8b140e2c9e19d"
Nov 03 18:26:05.927244 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:05.927214    5315 scope.go:122] "RemoveContainer" containerID="da9f37a8193cc773b4cae8f2459baa369721dedb302a26f5b52087473f086ca1"
Nov 03 18:26:19.090468 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:19.090427    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-r557b"]
Nov 03 18:26:19.090893 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:19.090827    5315 cpu_manager.go:414] "containerMap: removing container" podUID="8f7eb77f-2359-46f0-aa39-67dbff914c81" containerName="revoke-keys"
Nov 03 18:26:19.090893 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:19.090840    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f7eb77f-2359-46f0-aa39-67dbff914c81" containerName="revoke-keys"
Nov 03 18:26:19.090973 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:19.090913    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f7eb77f-2359-46f0-aa39-67dbff914c81" containerName="revoke-keys"
Nov 03 18:26:19.095633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:19.095582    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-r557b"
Nov 03 18:26:19.144170 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:19.144136    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:26:19.144170 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:19.144149    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:26:19.169365 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:19.169329    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ee1ca3d-1df0-408d-9c64-158a7e662e4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-r557b\" (UID: \"0ee1ca3d-1df0-408d-9c64-158a7e662e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-r557b"
Nov 03 18:26:19.169559 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:19.169402    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gcdp\" (UniqueName: \"kubernetes.io/projected/0ee1ca3d-1df0-408d-9c64-158a7e662e4d-kube-api-access-6gcdp\") pod \"maas-api-revoke-sub-e2fc9c5b-r557b\" (UID: \"0ee1ca3d-1df0-408d-9c64-158a7e662e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-r557b"
Nov 03 18:26:19.169675 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:19.169602    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ee1ca3d-1df0-408d-9c64-158a7e662e4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-r557b\" (UID: \"0ee1ca3d-1df0-408d-9c64-158a7e662e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-r557b"
Nov 03 18:26:19.190145 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:19.190108    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:26:19.190322 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:19.190279    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:26:19.192211 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:19.192181    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-r557b"]
Nov 03 18:26:19.271009 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:19.270971    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gcdp\" (UniqueName: \"kubernetes.io/projected/0ee1ca3d-1df0-408d-9c64-158a7e662e4d-kube-api-access-6gcdp\") pod \"maas-api-revoke-sub-e2fc9c5b-r557b\" (UID: \"0ee1ca3d-1df0-408d-9c64-158a7e662e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-r557b"
Nov 03 18:26:19.271218 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:19.271138    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ee1ca3d-1df0-408d-9c64-158a7e662e4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-r557b\" (UID: \"0ee1ca3d-1df0-408d-9c64-158a7e662e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-r557b"
Nov 03 18:26:19.271218 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:19.271191    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ee1ca3d-1df0-408d-9c64-158a7e662e4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-r557b\" (UID: \"0ee1ca3d-1df0-408d-9c64-158a7e662e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-r557b"
Nov 03 18:26:19.271791 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:19.271766    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ee1ca3d-1df0-408d-9c64-158a7e662e4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-r557b\" (UID: \"0ee1ca3d-1df0-408d-9c64-158a7e662e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-r557b"
Nov 03 18:26:19.273529 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:19.273511    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ee1ca3d-1df0-408d-9c64-158a7e662e4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-r557b\" (UID: \"0ee1ca3d-1df0-408d-9c64-158a7e662e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-r557b"
Nov 03 18:26:19.328071 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:19.328039    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gcdp\" (UniqueName: \"kubernetes.io/projected/0ee1ca3d-1df0-408d-9c64-158a7e662e4d-kube-api-access-6gcdp\") pod \"maas-api-revoke-sub-e2fc9c5b-r557b\" (UID: \"0ee1ca3d-1df0-408d-9c64-158a7e662e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-r557b"
Nov 03 18:26:19.408968 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:19.408881    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-r557b"
Nov 03 18:26:19.776970 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:19.776940    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-r557b"]
Nov 03 18:26:19.783520 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:26:19.783487    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ee1ca3d_1df0_408d_9c64_158a7e662e4d.slice/crio-0b29b16a142d00717d39c81d9b93eb8c2fd0c9cede59b68cac066d4f096c0185 WatchSource:0}: Error finding container 0b29b16a142d00717d39c81d9b93eb8c2fd0c9cede59b68cac066d4f096c0185: Status 404 returned error can't find the container with id 0b29b16a142d00717d39c81d9b93eb8c2fd0c9cede59b68cac066d4f096c0185
Nov 03 18:26:20.371171 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:20.371079    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-r557b" event={"ID":"0ee1ca3d-1df0-408d-9c64-158a7e662e4d","Type":"ContainerStarted","Data":"9623f8c329c41769ebcab81a008480baa381fa2669126d9da47b8d157d6f1e22"}
Nov 03 18:26:20.371171 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:20.371136    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-r557b" event={"ID":"0ee1ca3d-1df0-408d-9c64-158a7e662e4d","Type":"ContainerStarted","Data":"0b29b16a142d00717d39c81d9b93eb8c2fd0c9cede59b68cac066d4f096c0185"}
Nov 03 18:26:20.479092 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:20.479037    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-r557b" podStartSLOduration=1.479023977 podStartE2EDuration="1.479023977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:26:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:26:20.477912735 +0000 UTC m=+2419.494110803" watchObservedRunningTime="2026-11-03 18:26:20.479023977 +0000 UTC m=+2419.495222087"
Nov 03 18:26:21.377047 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:21.377008    5315 generic.go:356] "Generic (PLEG): container finished" podID="0ee1ca3d-1df0-408d-9c64-158a7e662e4d" containerID="9623f8c329c41769ebcab81a008480baa381fa2669126d9da47b8d157d6f1e22" exitCode=0
Nov 03 18:26:21.377513 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:21.377093    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-r557b" event={"ID":"0ee1ca3d-1df0-408d-9c64-158a7e662e4d","Type":"ContainerDied","Data":"9623f8c329c41769ebcab81a008480baa381fa2669126d9da47b8d157d6f1e22"}
Nov 03 18:26:22.520794 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:22.520762    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-r557b"
Nov 03 18:26:22.603664 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:22.603602    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ee1ca3d-1df0-408d-9c64-158a7e662e4d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ee1ca3d-1df0-408d-9c64-158a7e662e4d-maas-api-openshift-ca\") pod \"0ee1ca3d-1df0-408d-9c64-158a7e662e4d\" (UID: \"0ee1ca3d-1df0-408d-9c64-158a7e662e4d\") "
Nov 03 18:26:22.603852 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:22.603709    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ee1ca3d-1df0-408d-9c64-158a7e662e4d-kube-api-access-6gcdp\" (UniqueName: \"kubernetes.io/projected/0ee1ca3d-1df0-408d-9c64-158a7e662e4d-kube-api-access-6gcdp\") pod \"0ee1ca3d-1df0-408d-9c64-158a7e662e4d\" (UID: \"0ee1ca3d-1df0-408d-9c64-158a7e662e4d\") "
Nov 03 18:26:22.603852 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:22.603803    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ee1ca3d-1df0-408d-9c64-158a7e662e4d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ee1ca3d-1df0-408d-9c64-158a7e662e4d-maas-api-xks-ca\") pod \"0ee1ca3d-1df0-408d-9c64-158a7e662e4d\" (UID: \"0ee1ca3d-1df0-408d-9c64-158a7e662e4d\") "
Nov 03 18:26:22.604009 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:22.603986    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ee1ca3d-1df0-408d-9c64-158a7e662e4d-maas-api-openshift-ca" pod "0ee1ca3d-1df0-408d-9c64-158a7e662e4d" (UID: "0ee1ca3d-1df0-408d-9c64-158a7e662e4d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:26:22.604140 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:22.604121    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ee1ca3d-1df0-408d-9c64-158a7e662e4d-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:26:22.605859 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:22.605832    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ee1ca3d-1df0-408d-9c64-158a7e662e4d-maas-api-xks-ca" pod "0ee1ca3d-1df0-408d-9c64-158a7e662e4d" (UID: "0ee1ca3d-1df0-408d-9c64-158a7e662e4d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:26:22.605941 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:22.605927    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ee1ca3d-1df0-408d-9c64-158a7e662e4d-kube-api-access-6gcdp" pod "0ee1ca3d-1df0-408d-9c64-158a7e662e4d" (UID: "0ee1ca3d-1df0-408d-9c64-158a7e662e4d"). InnerVolumeSpecName "kube-api-access-6gcdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:26:22.705215 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:22.705127    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gcdp\" (UniqueName: \"kubernetes.io/projected/0ee1ca3d-1df0-408d-9c64-158a7e662e4d-kube-api-access-6gcdp\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:26:22.705215 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:22.705163    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ee1ca3d-1df0-408d-9c64-158a7e662e4d-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:26:23.388488 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:23.388407    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-r557b"
Nov 03 18:26:23.388488 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:23.388429    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-r557b" event={"ID":"0ee1ca3d-1df0-408d-9c64-158a7e662e4d","Type":"ContainerDied","Data":"0b29b16a142d00717d39c81d9b93eb8c2fd0c9cede59b68cac066d4f096c0185"}
Nov 03 18:26:23.388488 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:23.388483    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b29b16a142d00717d39c81d9b93eb8c2fd0c9cede59b68cac066d4f096c0185"
Nov 03 18:26:24.577043 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:26:24.577007    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:26:29.865218 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:29.865182    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-gttwx"]
Nov 03 18:26:29.865659 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:29.865548    5315 cpu_manager.go:414] "containerMap: removing container" podUID="0ee1ca3d-1df0-408d-9c64-158a7e662e4d" containerName="revoke-keys"
Nov 03 18:26:29.865659 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:29.865559    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ee1ca3d-1df0-408d-9c64-158a7e662e4d" containerName="revoke-keys"
Nov 03 18:26:29.865659 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:29.865637    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ee1ca3d-1df0-408d-9c64-158a7e662e4d" containerName="revoke-keys"
Nov 03 18:26:29.868740 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:29.868719    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-gttwx"
Nov 03 18:26:29.917570 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:29.917535    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:26:29.963570 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:29.963536    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:26:29.963570 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:29.963561    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:26:29.963786 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:29.963726    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:26:29.965391 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:29.965368    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-gttwx"]
Nov 03 18:26:29.969141 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:29.969115    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-gttwx\" (UID: \"ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-gttwx"
Nov 03 18:26:29.969274 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:29.969259    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-gttwx\" (UID: \"ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-gttwx"
Nov 03 18:26:29.969318 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:29.969291    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmmnz\" (UniqueName: \"kubernetes.io/projected/ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3-kube-api-access-gmmnz\") pod \"maas-api-revoke-sub-5a29efb3-gttwx\" (UID: \"ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-gttwx"
Nov 03 18:26:30.070010 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:30.069965    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-gttwx\" (UID: \"ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-gttwx"
Nov 03 18:26:30.070010 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:30.070015    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmmnz\" (UniqueName: \"kubernetes.io/projected/ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3-kube-api-access-gmmnz\") pod \"maas-api-revoke-sub-5a29efb3-gttwx\" (UID: \"ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-gttwx"
Nov 03 18:26:30.070242 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:30.070058    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-gttwx\" (UID: \"ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-gttwx"
Nov 03 18:26:30.070692 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:30.070670    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-gttwx\" (UID: \"ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-gttwx"
Nov 03 18:26:30.072504 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:30.072484    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-gttwx\" (UID: \"ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-gttwx"
Nov 03 18:26:30.127401 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:30.127328    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmmnz\" (UniqueName: \"kubernetes.io/projected/ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3-kube-api-access-gmmnz\") pod \"maas-api-revoke-sub-5a29efb3-gttwx\" (UID: \"ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-gttwx"
Nov 03 18:26:30.182524 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:30.182485    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-gttwx"
Nov 03 18:26:30.553508 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:30.553476    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-gttwx"]
Nov 03 18:26:30.559815 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:26:30.559784    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca5a2d37_4b8b_4b10_b5a0_d4edc802f0a3.slice/crio-863f9078563b0b5cb2620c6211c38d1af178c3d47816cf54bea01c3389f20530 WatchSource:0}: Error finding container 863f9078563b0b5cb2620c6211c38d1af178c3d47816cf54bea01c3389f20530: Status 404 returned error can't find the container with id 863f9078563b0b5cb2620c6211c38d1af178c3d47816cf54bea01c3389f20530
Nov 03 18:26:31.427290 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:31.427259    5315 generic.go:356] "Generic (PLEG): container finished" podID="ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3" containerID="7d3fdbc061fa247a304c2bb2b3036f633b3ece0b7ce29bdd90d64f9ab5611e4c" exitCode=0
Nov 03 18:26:31.427684 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:31.427300    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-gttwx" event={"ID":"ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3","Type":"ContainerDied","Data":"7d3fdbc061fa247a304c2bb2b3036f633b3ece0b7ce29bdd90d64f9ab5611e4c"}
Nov 03 18:26:31.427684 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:31.427340    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-gttwx" event={"ID":"ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3","Type":"ContainerStarted","Data":"863f9078563b0b5cb2620c6211c38d1af178c3d47816cf54bea01c3389f20530"}
Nov 03 18:26:32.625683 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:32.625643    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-gttwx"
Nov 03 18:26:32.694756 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:32.694651    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3-maas-api-openshift-ca\") pod \"ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3\" (UID: \"ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3\") "
Nov 03 18:26:32.694949 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:32.694792    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3-maas-api-xks-ca\") pod \"ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3\" (UID: \"ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3\") "
Nov 03 18:26:32.694949 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:32.694840    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3-kube-api-access-gmmnz\" (UniqueName: \"kubernetes.io/projected/ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3-kube-api-access-gmmnz\") pod \"ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3\" (UID: \"ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3\") "
Nov 03 18:26:32.695141 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:32.695108    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3-maas-api-openshift-ca" pod "ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3" (UID: "ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:26:32.697565 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:32.697518    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3-maas-api-xks-ca" pod "ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3" (UID: "ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:26:32.697730 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:32.697683    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3-kube-api-access-gmmnz" pod "ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3" (UID: "ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3"). InnerVolumeSpecName "kube-api-access-gmmnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:26:32.796179 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:32.796130    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:26:32.796179 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:32.796174    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:26:32.796463 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:32.796192    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmmnz\" (UniqueName: \"kubernetes.io/projected/ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3-kube-api-access-gmmnz\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:26:33.457695 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:33.452683    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-gttwx" event={"ID":"ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3","Type":"ContainerDied","Data":"863f9078563b0b5cb2620c6211c38d1af178c3d47816cf54bea01c3389f20530"}
Nov 03 18:26:33.457695 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:33.452789    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="863f9078563b0b5cb2620c6211c38d1af178c3d47816cf54bea01c3389f20530"
Nov 03 18:26:33.457695 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:33.452877    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-gttwx"
Nov 03 18:26:49.044026 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:49.043943    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tp6ht"]
Nov 03 18:26:49.089833 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:49.089794    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tp6ht"]
Nov 03 18:26:49.102862 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:49.102828    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5e59d9c-87a3-49d6-8efc-bb29567d789f" path="/var/lib/kubelet/pods/a5e59d9c-87a3-49d6-8efc-bb29567d789f/volumes"
Nov 03 18:26:50.678102 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:50.678058    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4x9x9"]
Nov 03 18:26:50.724024 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:50.723986    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4x9x9"]
Nov 03 18:26:50.800865 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:50.800829    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-48fgm"]
Nov 03 18:26:50.801279 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:50.801260    5315 cpu_manager.go:414] "containerMap: removing container" podUID="ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3" containerName="revoke-keys"
Nov 03 18:26:50.801373 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:50.801281    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3" containerName="revoke-keys"
Nov 03 18:26:50.801437 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:50.801401    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3" containerName="revoke-keys"
Nov 03 18:26:50.804507 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:50.804481    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-48fgm"
Nov 03 18:26:50.812432 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:50.812406    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85849579-f963-4e58-b330-eb25238efaec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-48fgm\" (UID: \"85849579-f963-4e58-b330-eb25238efaec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-48fgm"
Nov 03 18:26:50.812571 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:50.812449    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwwzx\" (UniqueName: \"kubernetes.io/projected/85849579-f963-4e58-b330-eb25238efaec-kube-api-access-nwwzx\") pod \"maas-api-revoke-sub-3d4156f1-48fgm\" (UID: \"85849579-f963-4e58-b330-eb25238efaec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-48fgm"
Nov 03 18:26:50.812571 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:50.812553    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85849579-f963-4e58-b330-eb25238efaec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-48fgm\" (UID: \"85849579-f963-4e58-b330-eb25238efaec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-48fgm"
Nov 03 18:26:50.853117 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:50.853085    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:26:50.853335 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:50.853310    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:26:50.853417 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:50.853357    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:26:50.853458 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:50.853422    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:26:50.865046 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:50.865016    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-48fgm"]
Nov 03 18:26:50.913057 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:50.913019    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85849579-f963-4e58-b330-eb25238efaec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-48fgm\" (UID: \"85849579-f963-4e58-b330-eb25238efaec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-48fgm"
Nov 03 18:26:50.913259 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:50.913074    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85849579-f963-4e58-b330-eb25238efaec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-48fgm\" (UID: \"85849579-f963-4e58-b330-eb25238efaec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-48fgm"
Nov 03 18:26:50.913259 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:50.913102    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwwzx\" (UniqueName: \"kubernetes.io/projected/85849579-f963-4e58-b330-eb25238efaec-kube-api-access-nwwzx\") pod \"maas-api-revoke-sub-3d4156f1-48fgm\" (UID: \"85849579-f963-4e58-b330-eb25238efaec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-48fgm"
Nov 03 18:26:50.913899 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:50.913863    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85849579-f963-4e58-b330-eb25238efaec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-48fgm\" (UID: \"85849579-f963-4e58-b330-eb25238efaec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-48fgm"
Nov 03 18:26:50.915691 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:50.915663    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85849579-f963-4e58-b330-eb25238efaec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-48fgm\" (UID: \"85849579-f963-4e58-b330-eb25238efaec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-48fgm"
Nov 03 18:26:50.970634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:50.970537    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwwzx\" (UniqueName: \"kubernetes.io/projected/85849579-f963-4e58-b330-eb25238efaec-kube-api-access-nwwzx\") pod \"maas-api-revoke-sub-3d4156f1-48fgm\" (UID: \"85849579-f963-4e58-b330-eb25238efaec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-48fgm"
Nov 03 18:26:51.104004 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:51.103968    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f7eb77f-2359-46f0-aa39-67dbff914c81" path="/var/lib/kubelet/pods/8f7eb77f-2359-46f0-aa39-67dbff914c81/volumes"
Nov 03 18:26:51.117513 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:51.117487    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-48fgm"
Nov 03 18:26:51.484453 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:51.484416    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-48fgm"]
Nov 03 18:26:51.491301 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:26:51.491268    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85849579_f963_4e58_b330_eb25238efaec.slice/crio-574d1b90cc51496081d91f4942d517ad7c782d4325cd4375b8c5b5c0998cc1e4 WatchSource:0}: Error finding container 574d1b90cc51496081d91f4942d517ad7c782d4325cd4375b8c5b5c0998cc1e4: Status 404 returned error can't find the container with id 574d1b90cc51496081d91f4942d517ad7c782d4325cd4375b8c5b5c0998cc1e4
Nov 03 18:26:51.547291 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:51.547257    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-48fgm" event={"ID":"85849579-f963-4e58-b330-eb25238efaec","Type":"ContainerStarted","Data":"574d1b90cc51496081d91f4942d517ad7c782d4325cd4375b8c5b5c0998cc1e4"}
Nov 03 18:26:52.553899 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:52.553861    5315 generic.go:356] "Generic (PLEG): container finished" podID="85849579-f963-4e58-b330-eb25238efaec" containerID="2d38c4c3c095eda7db19aa5715f1515dcfcd2e141470016968c035230a7db175" exitCode=0
Nov 03 18:26:52.554296 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:52.553946    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-48fgm" event={"ID":"85849579-f963-4e58-b330-eb25238efaec","Type":"ContainerDied","Data":"2d38c4c3c095eda7db19aa5715f1515dcfcd2e141470016968c035230a7db175"}
Nov 03 18:26:53.689888 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:53.689859    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-48fgm"
Nov 03 18:26:53.736926 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:53.736891    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85849579-f963-4e58-b330-eb25238efaec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85849579-f963-4e58-b330-eb25238efaec-maas-api-xks-ca\") pod \"85849579-f963-4e58-b330-eb25238efaec\" (UID: \"85849579-f963-4e58-b330-eb25238efaec\") "
Nov 03 18:26:53.737098 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:53.736940    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85849579-f963-4e58-b330-eb25238efaec-kube-api-access-nwwzx\" (UniqueName: \"kubernetes.io/projected/85849579-f963-4e58-b330-eb25238efaec-kube-api-access-nwwzx\") pod \"85849579-f963-4e58-b330-eb25238efaec\" (UID: \"85849579-f963-4e58-b330-eb25238efaec\") "
Nov 03 18:26:53.737098 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:53.737019    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85849579-f963-4e58-b330-eb25238efaec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85849579-f963-4e58-b330-eb25238efaec-maas-api-openshift-ca\") pod \"85849579-f963-4e58-b330-eb25238efaec\" (UID: \"85849579-f963-4e58-b330-eb25238efaec\") "
Nov 03 18:26:53.737387 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:53.737362    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85849579-f963-4e58-b330-eb25238efaec-maas-api-openshift-ca" pod "85849579-f963-4e58-b330-eb25238efaec" (UID: "85849579-f963-4e58-b330-eb25238efaec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:26:53.739063 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:53.739032    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85849579-f963-4e58-b330-eb25238efaec-maas-api-xks-ca" pod "85849579-f963-4e58-b330-eb25238efaec" (UID: "85849579-f963-4e58-b330-eb25238efaec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:26:53.739202 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:53.739181    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85849579-f963-4e58-b330-eb25238efaec-kube-api-access-nwwzx" pod "85849579-f963-4e58-b330-eb25238efaec" (UID: "85849579-f963-4e58-b330-eb25238efaec"). InnerVolumeSpecName "kube-api-access-nwwzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:26:53.838028 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:53.837933    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85849579-f963-4e58-b330-eb25238efaec-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:26:53.838028 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:53.837965    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwwzx\" (UniqueName: \"kubernetes.io/projected/85849579-f963-4e58-b330-eb25238efaec-kube-api-access-nwwzx\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:26:53.838028 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:53.837976    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85849579-f963-4e58-b330-eb25238efaec-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:26:54.564916 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:54.564877    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-48fgm"
Nov 03 18:26:54.565121 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:54.564871    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-48fgm" event={"ID":"85849579-f963-4e58-b330-eb25238efaec","Type":"ContainerDied","Data":"574d1b90cc51496081d91f4942d517ad7c782d4325cd4375b8c5b5c0998cc1e4"}
Nov 03 18:26:54.565121 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:26:54.565023    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="574d1b90cc51496081d91f4942d517ad7c782d4325cd4375b8c5b5c0998cc1e4"
Nov 03 18:26:54.578886 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:26:54.578842    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:27:05.954741 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:05.954710    5315 scope.go:122] "RemoveContainer" containerID="f817a122140fa1030af0b504f79bf1b433540d016ab69fd59ba88ffe1491b07b"
Nov 03 18:27:07.662763 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:07.662727    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj"]
Nov 03 18:27:07.663177 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:07.663113    5315 cpu_manager.go:414] "containerMap: removing container" podUID="85849579-f963-4e58-b330-eb25238efaec" containerName="revoke-keys"
Nov 03 18:27:07.663177 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:07.663125    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85849579-f963-4e58-b330-eb25238efaec" containerName="revoke-keys"
Nov 03 18:27:07.663260 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:07.663184    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="85849579-f963-4e58-b330-eb25238efaec" containerName="revoke-keys"
Nov 03 18:27:07.667727 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:07.667702    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj"
Nov 03 18:27:07.716369 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:07.716331    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-58e97a-openshift-default-dockercfg-rgtst\""
Nov 03 18:27:07.761921 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:07.761884    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q968\" (UniqueName: \"kubernetes.io/projected/7a198348-f996-4bab-8499-21d41c5c9bee-kube-api-access-2q968\") pod \"e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj\" (UID: \"7a198348-f996-4bab-8499-21d41c5c9bee\") " pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj"
Nov 03 18:27:07.762111 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:07.761941    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a198348-f996-4bab-8499-21d41c5c9bee-istiod-ca-cert\") pod \"e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj\" (UID: \"7a198348-f996-4bab-8499-21d41c5c9bee\") " pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj"
Nov 03 18:27:07.762111 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:07.762029    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a198348-f996-4bab-8499-21d41c5c9bee-workload-certs\") pod \"e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj\" (UID: \"7a198348-f996-4bab-8499-21d41c5c9bee\") " pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj"
Nov 03 18:27:07.762111 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:07.762089    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a198348-f996-4bab-8499-21d41c5c9bee-istio-data\") pod \"e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj\" (UID: \"7a198348-f996-4bab-8499-21d41c5c9bee\") " pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj"
Nov 03 18:27:07.762223 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:07.762121    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a198348-f996-4bab-8499-21d41c5c9bee-credential-socket\") pod \"e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj\" (UID: \"7a198348-f996-4bab-8499-21d41c5c9bee\") " pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj"
Nov 03 18:27:07.762223 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:07.762198    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a198348-f996-4bab-8499-21d41c5c9bee-istio-podinfo\") pod \"e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj\" (UID: \"7a198348-f996-4bab-8499-21d41c5c9bee\") " pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj"
Nov 03 18:27:07.762291 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:07.762233    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a198348-f996-4bab-8499-21d41c5c9bee-istio-token\") pod \"e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj\" (UID: \"7a198348-f996-4bab-8499-21d41c5c9bee\") " pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj"
Nov 03 18:27:07.762291 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:07.762251    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a198348-f996-4bab-8499-21d41c5c9bee-workload-socket\") pod \"e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj\" (UID: \"7a198348-f996-4bab-8499-21d41c5c9bee\") " pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj"
Nov 03 18:27:07.762354 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:07.762295    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a198348-f996-4bab-8499-21d41c5c9bee-istio-envoy\") pod \"e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj\" (UID: \"7a198348-f996-4bab-8499-21d41c5c9bee\") " pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj"
Nov 03 18:27:07.776470 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:07.776441    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj"]
Nov 03 18:27:07.863006 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:07.862953    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a198348-f996-4bab-8499-21d41c5c9bee-istio-podinfo\") pod \"e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj\" (UID: \"7a198348-f996-4bab-8499-21d41c5c9bee\") " pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj"
Nov 03 18:27:07.863217 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:07.863012    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a198348-f996-4bab-8499-21d41c5c9bee-istio-token\") pod \"e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj\" (UID: \"7a198348-f996-4bab-8499-21d41c5c9bee\") " pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj"
Nov 03 18:27:07.863217 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:07.863040    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a198348-f996-4bab-8499-21d41c5c9bee-workload-socket\") pod \"e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj\" (UID: \"7a198348-f996-4bab-8499-21d41c5c9bee\") " pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj"
Nov 03 18:27:07.863217 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:07.863075    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a198348-f996-4bab-8499-21d41c5c9bee-istio-envoy\") pod \"e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj\" (UID: \"7a198348-f996-4bab-8499-21d41c5c9bee\") " pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj"
Nov 03 18:27:07.863217 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:07.863129    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q968\" (UniqueName: \"kubernetes.io/projected/7a198348-f996-4bab-8499-21d41c5c9bee-kube-api-access-2q968\") pod \"e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj\" (UID: \"7a198348-f996-4bab-8499-21d41c5c9bee\") " pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj"
Nov 03 18:27:07.863217 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:07.863198    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a198348-f996-4bab-8499-21d41c5c9bee-istiod-ca-cert\") pod \"e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj\" (UID: \"7a198348-f996-4bab-8499-21d41c5c9bee\") " pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj"
Nov 03 18:27:07.863565 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:07.863227    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a198348-f996-4bab-8499-21d41c5c9bee-workload-certs\") pod \"e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj\" (UID: \"7a198348-f996-4bab-8499-21d41c5c9bee\") " pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj"
Nov 03 18:27:07.863565 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:07.863287    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a198348-f996-4bab-8499-21d41c5c9bee-istio-data\") pod \"e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj\" (UID: \"7a198348-f996-4bab-8499-21d41c5c9bee\") " pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj"
Nov 03 18:27:07.863565 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:07.863324    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a198348-f996-4bab-8499-21d41c5c9bee-credential-socket\") pod \"e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj\" (UID: \"7a198348-f996-4bab-8499-21d41c5c9bee\") " pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj"
Nov 03 18:27:07.864163 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:07.864114    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a198348-f996-4bab-8499-21d41c5c9bee-credential-socket\") pod \"e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj\" (UID: \"7a198348-f996-4bab-8499-21d41c5c9bee\") " pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj"
Nov 03 18:27:07.864163 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:07.864134    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a198348-f996-4bab-8499-21d41c5c9bee-workload-socket\") pod \"e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj\" (UID: \"7a198348-f996-4bab-8499-21d41c5c9bee\") " pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj"
Nov 03 18:27:07.865508 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:07.864509    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a198348-f996-4bab-8499-21d41c5c9bee-istio-data\") pod \"e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj\" (UID: \"7a198348-f996-4bab-8499-21d41c5c9bee\") " pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj"
Nov 03 18:27:07.865508 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:07.865064    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a198348-f996-4bab-8499-21d41c5c9bee-istiod-ca-cert\") pod \"e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj\" (UID: \"7a198348-f996-4bab-8499-21d41c5c9bee\") " pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj"
Nov 03 18:27:07.865508 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:07.865330    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a198348-f996-4bab-8499-21d41c5c9bee-workload-certs\") pod \"e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj\" (UID: \"7a198348-f996-4bab-8499-21d41c5c9bee\") " pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj"
Nov 03 18:27:07.869049 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:07.869020    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a198348-f996-4bab-8499-21d41c5c9bee-istio-envoy\") pod \"e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj\" (UID: \"7a198348-f996-4bab-8499-21d41c5c9bee\") " pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj"
Nov 03 18:27:07.869407 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:07.869373    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a198348-f996-4bab-8499-21d41c5c9bee-istio-podinfo\") pod \"e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj\" (UID: \"7a198348-f996-4bab-8499-21d41c5c9bee\") " pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj"
Nov 03 18:27:07.920954 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:07.920882    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a198348-f996-4bab-8499-21d41c5c9bee-istio-token\") pod \"e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj\" (UID: \"7a198348-f996-4bab-8499-21d41c5c9bee\") " pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj"
Nov 03 18:27:07.922548 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:07.922519    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q968\" (UniqueName: \"kubernetes.io/projected/7a198348-f996-4bab-8499-21d41c5c9bee-kube-api-access-2q968\") pod \"e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj\" (UID: \"7a198348-f996-4bab-8499-21d41c5c9bee\") " pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj"
Nov 03 18:27:07.980920 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:07.980882    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj"
Nov 03 18:27:08.403806 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:08.403772    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj"]
Nov 03 18:27:08.410359 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:27:08.410332    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a198348_f996_4bab_8499_21d41c5c9bee.slice/crio-776214ce4bc652330c9845901f603a332e2e41497da088d9d2be01298bc68d1a WatchSource:0}: Error finding container 776214ce4bc652330c9845901f603a332e2e41497da088d9d2be01298bc68d1a: Status 404 returned error can't find the container with id 776214ce4bc652330c9845901f603a332e2e41497da088d9d2be01298bc68d1a
Nov 03 18:27:08.412420 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:08.412388    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:27:08.412481 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:08.412454    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:27:08.412521 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:08.412483    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:27:08.633161 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:08.633125    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj" event={"ID":"7a198348-f996-4bab-8499-21d41c5c9bee","Type":"ContainerStarted","Data":"75019fa6048002a549cc9c9b6e0ea4f03e673173285b36f22d5cdba173a5dccb"}
Nov 03 18:27:08.633354 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:08.633192    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj" event={"ID":"7a198348-f996-4bab-8499-21d41c5c9bee","Type":"ContainerStarted","Data":"776214ce4bc652330c9845901f603a332e2e41497da088d9d2be01298bc68d1a"}
Nov 03 18:27:08.747533 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:08.747413    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj" podStartSLOduration=1.747400182 podStartE2EDuration="1.747400182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:27:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:27:08.745477948 +0000 UTC m=+2467.761676017" watchObservedRunningTime="2026-11-03 18:27:08.747400182 +0000 UTC m=+2467.763598250"
Nov 03 18:27:08.981473 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:08.981436    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj"
Nov 03 18:27:09.986155 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:09.986124    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj"
Nov 03 18:27:10.642903 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:10.642867    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj"
Nov 03 18:27:10.643981 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:10.643961    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj"
Nov 03 18:27:22.045661 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:22.045618    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7bg4x"]
Nov 03 18:27:22.091584 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:22.091545    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7bg4x"]
Nov 03 18:27:23.102374 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:23.102336    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9094aa8c-6312-4f34-8636-a0b4a5c7c103" path="/var/lib/kubelet/pods/9094aa8c-6312-4f34-8636-a0b4a5c7c103/volumes"
Nov 03 18:27:24.577000 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:27:24.576962    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:27:34.053430 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:34.053390    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7mt6j"]
Nov 03 18:27:34.099507 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:34.099458    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7mt6j"]
Nov 03 18:27:35.102886 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:35.102841    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce4b89d6-a516-4513-8a2a-6e04be561d17" path="/var/lib/kubelet/pods/ce4b89d6-a516-4513-8a2a-6e04be561d17/volumes"
Nov 03 18:27:36.789737 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:36.789700    5315 generic.go:356] "Generic (PLEG): container finished" podID="7a198348-f996-4bab-8499-21d41c5c9bee" containerID="75019fa6048002a549cc9c9b6e0ea4f03e673173285b36f22d5cdba173a5dccb" exitCode=0
Nov 03 18:27:36.790209 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:36.789775    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj" event={"ID":"7a198348-f996-4bab-8499-21d41c5c9bee","Type":"ContainerDied","Data":"75019fa6048002a549cc9c9b6e0ea4f03e673173285b36f22d5cdba173a5dccb"}
Nov 03 18:27:36.790209 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:36.790174    5315 scope.go:122] "RemoveContainer" containerID="75019fa6048002a549cc9c9b6e0ea4f03e673173285b36f22d5cdba173a5dccb"
Nov 03 18:27:36.791324 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:36.791295    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:27:36.791436 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:36.791361    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:27:36.791436 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:36.791395    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:27:37.796263 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:37.796227    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj" event={"ID":"7a198348-f996-4bab-8499-21d41c5c9bee","Type":"ContainerStarted","Data":"c5000c8a4ad3b196d676fae0fb3df3ce4e822d52dba2a4a62db37a9055ad3ee7"}
Nov 03 18:27:37.981974 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:37.981925    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj"
Nov 03 18:27:37.983119 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:37.983090    5315 patch_prober.go:29] interesting pod/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body=
Nov 03 18:27:37.983212 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:37.983158    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj" podUID="7a198348-f996-4bab-8499-21d41c5c9bee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused"
Nov 03 18:27:38.981519 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:38.981480    5315 patch_prober.go:29] interesting pod/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body=
Nov 03 18:27:38.982050 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:38.981544    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj" podUID="7a198348-f996-4bab-8499-21d41c5c9bee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused"
Nov 03 18:27:39.867245 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:39.867209    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-58e97a/simulator-single-main-58e97a-kserve-586d66fb65-svntp"]
Nov 03 18:27:39.870239 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:39.870218    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-58e97a/simulator-single-main-58e97a-kserve-586d66fb65-svntp"
Nov 03 18:27:39.951204 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:39.951158    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21fb225c-08c4-443b-bde8-76b22a1a2aa0-model-cache\") pod \"simulator-single-main-58e97a-kserve-586d66fb65-svntp\" (UID: \"21fb225c-08c4-443b-bde8-76b22a1a2aa0\") " pod="e2e-models-e2e-worker-single-main-58e97a/simulator-single-main-58e97a-kserve-586d66fb65-svntp"
Nov 03 18:27:39.951413 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:39.951217    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21fb225c-08c4-443b-bde8-76b22a1a2aa0-tmp-dir\") pod \"simulator-single-main-58e97a-kserve-586d66fb65-svntp\" (UID: \"21fb225c-08c4-443b-bde8-76b22a1a2aa0\") " pod="e2e-models-e2e-worker-single-main-58e97a/simulator-single-main-58e97a-kserve-586d66fb65-svntp"
Nov 03 18:27:39.951413 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:39.951383    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21fb225c-08c4-443b-bde8-76b22a1a2aa0-home\") pod \"simulator-single-main-58e97a-kserve-586d66fb65-svntp\" (UID: \"21fb225c-08c4-443b-bde8-76b22a1a2aa0\") " pod="e2e-models-e2e-worker-single-main-58e97a/simulator-single-main-58e97a-kserve-586d66fb65-svntp"
Nov 03 18:27:39.951561 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:39.951439    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcljg\" (UniqueName: \"kubernetes.io/projected/21fb225c-08c4-443b-bde8-76b22a1a2aa0-kube-api-access-jcljg\") pod \"simulator-single-main-58e97a-kserve-586d66fb65-svntp\" (UID: \"21fb225c-08c4-443b-bde8-76b22a1a2aa0\") " pod="e2e-models-e2e-worker-single-main-58e97a/simulator-single-main-58e97a-kserve-586d66fb65-svntp"
Nov 03 18:27:39.951561 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:39.951470    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21fb225c-08c4-443b-bde8-76b22a1a2aa0-tls-certs\") pod \"simulator-single-main-58e97a-kserve-586d66fb65-svntp\" (UID: \"21fb225c-08c4-443b-bde8-76b22a1a2aa0\") " pod="e2e-models-e2e-worker-single-main-58e97a/simulator-single-main-58e97a-kserve-586d66fb65-svntp"
Nov 03 18:27:39.951561 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:39.951540    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21fb225c-08c4-443b-bde8-76b22a1a2aa0-dshm\") pod \"simulator-single-main-58e97a-kserve-586d66fb65-svntp\" (UID: \"21fb225c-08c4-443b-bde8-76b22a1a2aa0\") " pod="e2e-models-e2e-worker-single-main-58e97a/simulator-single-main-58e97a-kserve-586d66fb65-svntp"
Nov 03 18:27:39.964680 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:39.964648    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-58e97a\"/\"openshift-service-ca.crt\""
Nov 03 18:27:39.964851 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:39.964756    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-58e97a\"/\"kube-root-ca.crt\""
Nov 03 18:27:39.964851 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:39.964751    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-58e97a\"/\"simulator-single-main-58e97a-kserve-self-signed-certs\""
Nov 03 18:27:39.964851 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:39.964843    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-58e97a\"/\"default-dockercfg-cbg85\""
Nov 03 18:27:39.966718 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:39.966700    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-58e97a/simulator-single-main-58e97a-kserve-586d66fb65-svntp"]
Nov 03 18:27:39.981440 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:39.981410    5315 patch_prober.go:29] interesting pod/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body=
Nov 03 18:27:39.981696 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:39.981661    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj" podUID="7a198348-f996-4bab-8499-21d41c5c9bee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused"
Nov 03 18:27:40.054049 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:40.054001    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21fb225c-08c4-443b-bde8-76b22a1a2aa0-dshm\") pod \"simulator-single-main-58e97a-kserve-586d66fb65-svntp\" (UID: \"21fb225c-08c4-443b-bde8-76b22a1a2aa0\") " pod="e2e-models-e2e-worker-single-main-58e97a/simulator-single-main-58e97a-kserve-586d66fb65-svntp"
Nov 03 18:27:40.054269 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:40.054081    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21fb225c-08c4-443b-bde8-76b22a1a2aa0-model-cache\") pod \"simulator-single-main-58e97a-kserve-586d66fb65-svntp\" (UID: \"21fb225c-08c4-443b-bde8-76b22a1a2aa0\") " pod="e2e-models-e2e-worker-single-main-58e97a/simulator-single-main-58e97a-kserve-586d66fb65-svntp"
Nov 03 18:27:40.054269 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:40.054102    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21fb225c-08c4-443b-bde8-76b22a1a2aa0-tmp-dir\") pod \"simulator-single-main-58e97a-kserve-586d66fb65-svntp\" (UID: \"21fb225c-08c4-443b-bde8-76b22a1a2aa0\") " pod="e2e-models-e2e-worker-single-main-58e97a/simulator-single-main-58e97a-kserve-586d66fb65-svntp"
Nov 03 18:27:40.054269 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:40.054133    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21fb225c-08c4-443b-bde8-76b22a1a2aa0-home\") pod \"simulator-single-main-58e97a-kserve-586d66fb65-svntp\" (UID: \"21fb225c-08c4-443b-bde8-76b22a1a2aa0\") " pod="e2e-models-e2e-worker-single-main-58e97a/simulator-single-main-58e97a-kserve-586d66fb65-svntp"
Nov 03 18:27:40.054269 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:40.054153    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcljg\" (UniqueName: \"kubernetes.io/projected/21fb225c-08c4-443b-bde8-76b22a1a2aa0-kube-api-access-jcljg\") pod \"simulator-single-main-58e97a-kserve-586d66fb65-svntp\" (UID: \"21fb225c-08c4-443b-bde8-76b22a1a2aa0\") " pod="e2e-models-e2e-worker-single-main-58e97a/simulator-single-main-58e97a-kserve-586d66fb65-svntp"
Nov 03 18:27:40.054269 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:40.054168    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21fb225c-08c4-443b-bde8-76b22a1a2aa0-tls-certs\") pod \"simulator-single-main-58e97a-kserve-586d66fb65-svntp\" (UID: \"21fb225c-08c4-443b-bde8-76b22a1a2aa0\") " pod="e2e-models-e2e-worker-single-main-58e97a/simulator-single-main-58e97a-kserve-586d66fb65-svntp"
Nov 03 18:27:40.054980 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:40.054958    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21fb225c-08c4-443b-bde8-76b22a1a2aa0-home\") pod \"simulator-single-main-58e97a-kserve-586d66fb65-svntp\" (UID: \"21fb225c-08c4-443b-bde8-76b22a1a2aa0\") " pod="e2e-models-e2e-worker-single-main-58e97a/simulator-single-main-58e97a-kserve-586d66fb65-svntp"
Nov 03 18:27:40.055655 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:40.055624    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21fb225c-08c4-443b-bde8-76b22a1a2aa0-model-cache\") pod \"simulator-single-main-58e97a-kserve-586d66fb65-svntp\" (UID: \"21fb225c-08c4-443b-bde8-76b22a1a2aa0\") " pod="e2e-models-e2e-worker-single-main-58e97a/simulator-single-main-58e97a-kserve-586d66fb65-svntp"
Nov 03 18:27:40.056020 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:40.055997    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21fb225c-08c4-443b-bde8-76b22a1a2aa0-tmp-dir\") pod \"simulator-single-main-58e97a-kserve-586d66fb65-svntp\" (UID: \"21fb225c-08c4-443b-bde8-76b22a1a2aa0\") " pod="e2e-models-e2e-worker-single-main-58e97a/simulator-single-main-58e97a-kserve-586d66fb65-svntp"
Nov 03 18:27:40.058082 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:40.058058    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21fb225c-08c4-443b-bde8-76b22a1a2aa0-tls-certs\") pod \"simulator-single-main-58e97a-kserve-586d66fb65-svntp\" (UID: \"21fb225c-08c4-443b-bde8-76b22a1a2aa0\") " pod="e2e-models-e2e-worker-single-main-58e97a/simulator-single-main-58e97a-kserve-586d66fb65-svntp"
Nov 03 18:27:40.059013 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:40.058992    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21fb225c-08c4-443b-bde8-76b22a1a2aa0-dshm\") pod \"simulator-single-main-58e97a-kserve-586d66fb65-svntp\" (UID: \"21fb225c-08c4-443b-bde8-76b22a1a2aa0\") " pod="e2e-models-e2e-worker-single-main-58e97a/simulator-single-main-58e97a-kserve-586d66fb65-svntp"
Nov 03 18:27:40.112037 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:40.112001    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcljg\" (UniqueName: \"kubernetes.io/projected/21fb225c-08c4-443b-bde8-76b22a1a2aa0-kube-api-access-jcljg\") pod \"simulator-single-main-58e97a-kserve-586d66fb65-svntp\" (UID: \"21fb225c-08c4-443b-bde8-76b22a1a2aa0\") " pod="e2e-models-e2e-worker-single-main-58e97a/simulator-single-main-58e97a-kserve-586d66fb65-svntp"
Nov 03 18:27:40.180810 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:40.180732    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-58e97a/simulator-single-main-58e97a-kserve-586d66fb65-svntp"
Nov 03 18:27:40.573839 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:40.573797    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-58e97a/simulator-single-main-58e97a-kserve-586d66fb65-svntp"]
Nov 03 18:27:40.580048 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:27:40.580013    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21fb225c_08c4_443b_bde8_76b22a1a2aa0.slice/crio-03f69ee1bb1cc28e3924a80e4fcd64ee3bc0174caaba4bba6b75de0641d1eb46 WatchSource:0}: Error finding container 03f69ee1bb1cc28e3924a80e4fcd64ee3bc0174caaba4bba6b75de0641d1eb46: Status 404 returned error can't find the container with id 03f69ee1bb1cc28e3924a80e4fcd64ee3bc0174caaba4bba6b75de0641d1eb46
Nov 03 18:27:40.645882 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:40.645841    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj"
Nov 03 18:27:40.818243 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:40.818204    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-58e97a/simulator-single-main-58e97a-kserve-586d66fb65-svntp" event={"ID":"21fb225c-08c4-443b-bde8-76b22a1a2aa0","Type":"ContainerStarted","Data":"13c652dd6a0a39ba3b8bd9656cdfff6264235716d10acc58395dbd0de60b87c2"}
Nov 03 18:27:40.818243 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:40.818252    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-58e97a/simulator-single-main-58e97a-kserve-586d66fb65-svntp" event={"ID":"21fb225c-08c4-443b-bde8-76b22a1a2aa0","Type":"ContainerStarted","Data":"03f69ee1bb1cc28e3924a80e4fcd64ee3bc0174caaba4bba6b75de0641d1eb46"}
Nov 03 18:27:40.981995 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:40.981887    5315 patch_prober.go:29] interesting pod/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body=
Nov 03 18:27:40.981995 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:40.981952    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj" podUID="7a198348-f996-4bab-8499-21d41c5c9bee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused"
Nov 03 18:27:41.181067 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:41.181030    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-58e97a/simulator-single-main-58e97a-kserve-586d66fb65-svntp"
Nov 03 18:27:41.981717 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:41.981681    5315 patch_prober.go:29] interesting pod/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body=
Nov 03 18:27:41.981915 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:41.981736    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj" podUID="7a198348-f996-4bab-8499-21d41c5c9bee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused"
Nov 03 18:27:42.985469 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:42.985439    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj"
Nov 03 18:27:42.986351 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:42.986331    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj"
Nov 03 18:27:43.145084 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:43.145027    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-58e97a/simulator-single-main-58e97a-kserve-586d66fb65-svntp" podStartSLOduration=4.145012486 podStartE2EDuration="4.145012486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:27:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:27:40.947818102 +0000 UTC m=+2499.964016172" watchObservedRunningTime="2026-11-03 18:27:43.145012486 +0000 UTC m=+2502.161210554"
Nov 03 18:27:50.181507 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:50.181463    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-58e97a/simulator-single-main-58e97a-kserve-586d66fb65-svntp"
Nov 03 18:27:50.194424 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:50.194391    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-58e97a/simulator-single-main-58e97a-kserve-586d66fb65-svntp"
Nov 03 18:27:50.878669 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:27:50.878639    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-58e97a/simulator-single-main-58e97a-kserve-586d66fb65-svntp"
Nov 03 18:27:54.576839 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:27:54.576797    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:28:05.975573 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:05.975539    5315 scope.go:122] "RemoveContainer" containerID="f4ac6009772f327b65c2afdb1172cfb8c8c13abe2c9707f3b19276c5c1ed4669"
Nov 03 18:28:09.152423 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:09.152384    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vgw55"]
Nov 03 18:28:09.198204 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:09.198168    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vgw55"]
Nov 03 18:28:11.102804 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:11.102762    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0625122c-faa5-4942-b9e3-c764b52e4a1b" path="/var/lib/kubelet/pods/0625122c-faa5-4942-b9e3-c764b52e4a1b/volumes"
Nov 03 18:28:20.045697 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:20.045603    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wzbdd"]
Nov 03 18:28:20.091689 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:20.091656    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wzbdd"]
Nov 03 18:28:21.104022 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:21.103980    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a29bc978-93c9-4602-88c3-ceefba86050b" path="/var/lib/kubelet/pods/a29bc978-93c9-4602-88c3-ceefba86050b/volumes"
Nov 03 18:28:24.577066 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:28:24.577028    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:28:24.981664 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:24.981557    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f68c122-mct4t"]
Nov 03 18:28:24.986732 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:24.986705    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f68c122-mct4t"
Nov 03 18:28:25.081427 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:25.081384    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:28:25.081427 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:25.081398    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:28:25.081691 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:25.081403    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:28:25.081691 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:25.081403    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:28:25.083139 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:25.083119    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f68c122-mct4t"]
Nov 03 18:28:25.087785 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:25.087766    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f68c122-mct4t\" (UID: \"bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f68c122-mct4t"
Nov 03 18:28:25.087855 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:25.087809    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f68c122-mct4t\" (UID: \"bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f68c122-mct4t"
Nov 03 18:28:25.087855 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:25.087827    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztcl5\" (UniqueName: \"kubernetes.io/projected/bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb-kube-api-access-ztcl5\") pod \"maas-api-revoke-sub-6f68c122-mct4t\" (UID: \"bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f68c122-mct4t"
Nov 03 18:28:25.188814 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:25.188775    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f68c122-mct4t\" (UID: \"bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f68c122-mct4t"
Nov 03 18:28:25.188814 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:25.188812    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztcl5\" (UniqueName: \"kubernetes.io/projected/bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb-kube-api-access-ztcl5\") pod \"maas-api-revoke-sub-6f68c122-mct4t\" (UID: \"bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f68c122-mct4t"
Nov 03 18:28:25.189084 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:25.188893    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f68c122-mct4t\" (UID: \"bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f68c122-mct4t"
Nov 03 18:28:25.189497 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:25.189465    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f68c122-mct4t\" (UID: \"bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f68c122-mct4t"
Nov 03 18:28:25.191246 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:25.191222    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f68c122-mct4t\" (UID: \"bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f68c122-mct4t"
Nov 03 18:28:25.246602 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:25.246527    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztcl5\" (UniqueName: \"kubernetes.io/projected/bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb-kube-api-access-ztcl5\") pod \"maas-api-revoke-sub-6f68c122-mct4t\" (UID: \"bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f68c122-mct4t"
Nov 03 18:28:25.299535 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:25.299495    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f68c122-mct4t"
Nov 03 18:28:25.674182 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:25.674139    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f68c122-mct4t"]
Nov 03 18:28:25.681513 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:28:25.681474    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd908b86_7972_4b0a_8ce1_e61fc0ae1fdb.slice/crio-19b868d6d664db792c217694c0d18437915af0ae4cf30f493ffc955a7a998f52 WatchSource:0}: Error finding container 19b868d6d664db792c217694c0d18437915af0ae4cf30f493ffc955a7a998f52: Status 404 returned error can't find the container with id 19b868d6d664db792c217694c0d18437915af0ae4cf30f493ffc955a7a998f52
Nov 03 18:28:26.052554 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:26.052519    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f68c122-mct4t" event={"ID":"bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb","Type":"ContainerStarted","Data":"6c674a95cc949138c940c9984b541fc6716191aff24f5a0ab53794ce3c972d9f"}
Nov 03 18:28:26.052779 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:26.052577    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f68c122-mct4t" event={"ID":"bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb","Type":"ContainerStarted","Data":"19b868d6d664db792c217694c0d18437915af0ae4cf30f493ffc955a7a998f52"}
Nov 03 18:28:26.160710 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:26.160653    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f68c122-mct4t" podStartSLOduration=2.160636832 podStartE2EDuration="2.160636832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:28:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:28:26.159646473 +0000 UTC m=+2545.175844538" watchObservedRunningTime="2026-11-03 18:28:26.160636832 +0000 UTC m=+2545.176834893"
Nov 03 18:28:27.059490 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:27.059454    5315 generic.go:356] "Generic (PLEG): container finished" podID="bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb" containerID="6c674a95cc949138c940c9984b541fc6716191aff24f5a0ab53794ce3c972d9f" exitCode=0
Nov 03 18:28:27.059926 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:27.059535    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f68c122-mct4t" event={"ID":"bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb","Type":"ContainerDied","Data":"6c674a95cc949138c940c9984b541fc6716191aff24f5a0ab53794ce3c972d9f"}
Nov 03 18:28:28.195652 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:28.195625    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f68c122-mct4t"
Nov 03 18:28:28.226909 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:28.226878    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb-kube-api-access-ztcl5\" (UniqueName: \"kubernetes.io/projected/bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb-kube-api-access-ztcl5\") pod \"bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb\" (UID: \"bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb\") "
Nov 03 18:28:28.227046 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:28.226934    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb-maas-api-openshift-ca\") pod \"bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb\" (UID: \"bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb\") "
Nov 03 18:28:28.227130 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:28.227111    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb-maas-api-xks-ca\") pod \"bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb\" (UID: \"bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb\") "
Nov 03 18:28:28.227295 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:28.227273    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb-maas-api-openshift-ca" pod "bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb" (UID: "bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:28:28.227462 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:28.227449    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:28:28.229034 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:28.229008    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb-kube-api-access-ztcl5" pod "bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb" (UID: "bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb"). InnerVolumeSpecName "kube-api-access-ztcl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:28:28.229133 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:28.229048    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb-maas-api-xks-ca" pod "bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb" (UID: "bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:28:28.329073 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:28.328968    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztcl5\" (UniqueName: \"kubernetes.io/projected/bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb-kube-api-access-ztcl5\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:28:28.329073 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:28.329029    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:28:29.077820 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:29.077784    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f68c122-mct4t" event={"ID":"bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb","Type":"ContainerDied","Data":"19b868d6d664db792c217694c0d18437915af0ae4cf30f493ffc955a7a998f52"}
Nov 03 18:28:29.077820 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:29.077808    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f68c122-mct4t"
Nov 03 18:28:29.078069 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:29.077864    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19b868d6d664db792c217694c0d18437915af0ae4cf30f493ffc955a7a998f52"
Nov 03 18:28:37.198101 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:37.198055    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-58e97a-d97mn"]
Nov 03 18:28:37.198686 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:37.198668    5315 cpu_manager.go:414] "containerMap: removing container" podUID="bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb" containerName="revoke-keys"
Nov 03 18:28:37.198771 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:37.198690    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb" containerName="revoke-keys"
Nov 03 18:28:37.198842 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:37.198824    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb" containerName="revoke-keys"
Nov 03 18:28:37.202113 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:37.202089    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-58e97a-d97mn"
Nov 03 18:28:37.250767 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:37.250735    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:28:37.296887 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:37.296851    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:28:37.296887 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:37.296857    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:28:37.297094 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:37.296951    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:28:37.298792 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:37.298770    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-58e97a-d97mn"]
Nov 03 18:28:37.318441 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:37.318400    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mlkq\" (UniqueName: \"kubernetes.io/projected/c9d84bbc-3809-4942-9cf5-06038a7d0611-kube-api-access-6mlkq\") pod \"maas-api-revoke-keys-e2e-worker-single-main-58e97a-d97mn\" (UID: \"c9d84bbc-3809-4942-9cf5-06038a7d0611\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-58e97a-d97mn"
Nov 03 18:28:37.318659 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:37.318453    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9d84bbc-3809-4942-9cf5-06038a7d0611-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-58e97a-d97mn\" (UID: \"c9d84bbc-3809-4942-9cf5-06038a7d0611\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-58e97a-d97mn"
Nov 03 18:28:37.318659 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:37.318471    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9d84bbc-3809-4942-9cf5-06038a7d0611-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-58e97a-d97mn\" (UID: \"c9d84bbc-3809-4942-9cf5-06038a7d0611\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-58e97a-d97mn"
Nov 03 18:28:37.419547 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:37.419506    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mlkq\" (UniqueName: \"kubernetes.io/projected/c9d84bbc-3809-4942-9cf5-06038a7d0611-kube-api-access-6mlkq\") pod \"maas-api-revoke-keys-e2e-worker-single-main-58e97a-d97mn\" (UID: \"c9d84bbc-3809-4942-9cf5-06038a7d0611\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-58e97a-d97mn"
Nov 03 18:28:37.419771 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:37.419564    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9d84bbc-3809-4942-9cf5-06038a7d0611-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-58e97a-d97mn\" (UID: \"c9d84bbc-3809-4942-9cf5-06038a7d0611\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-58e97a-d97mn"
Nov 03 18:28:37.419771 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:37.419593    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9d84bbc-3809-4942-9cf5-06038a7d0611-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-58e97a-d97mn\" (UID: \"c9d84bbc-3809-4942-9cf5-06038a7d0611\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-58e97a-d97mn"
Nov 03 18:28:37.420250 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:37.420226    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9d84bbc-3809-4942-9cf5-06038a7d0611-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-58e97a-d97mn\" (UID: \"c9d84bbc-3809-4942-9cf5-06038a7d0611\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-58e97a-d97mn"
Nov 03 18:28:37.422026 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:37.422003    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9d84bbc-3809-4942-9cf5-06038a7d0611-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-58e97a-d97mn\" (UID: \"c9d84bbc-3809-4942-9cf5-06038a7d0611\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-58e97a-d97mn"
Nov 03 18:28:37.476748 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:37.476663    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mlkq\" (UniqueName: \"kubernetes.io/projected/c9d84bbc-3809-4942-9cf5-06038a7d0611-kube-api-access-6mlkq\") pod \"maas-api-revoke-keys-e2e-worker-single-main-58e97a-d97mn\" (UID: \"c9d84bbc-3809-4942-9cf5-06038a7d0611\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-58e97a-d97mn"
Nov 03 18:28:37.514702 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:37.514668    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-58e97a-d97mn"
Nov 03 18:28:37.883081 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:37.883043    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-58e97a-d97mn"]
Nov 03 18:28:37.892114 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:28:37.892085    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9d84bbc_3809_4942_9cf5_06038a7d0611.slice/crio-e96eb514d27273962d56c48f828280f82e0b741503001d29741cacdcdc67c013 WatchSource:0}: Error finding container e96eb514d27273962d56c48f828280f82e0b741503001d29741cacdcdc67c013: Status 404 returned error can't find the container with id e96eb514d27273962d56c48f828280f82e0b741503001d29741cacdcdc67c013
Nov 03 18:28:38.124343 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:38.124312    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-58e97a-d97mn" event={"ID":"c9d84bbc-3809-4942-9cf5-06038a7d0611","Type":"ContainerStarted","Data":"e96eb514d27273962d56c48f828280f82e0b741503001d29741cacdcdc67c013"}
Nov 03 18:28:39.130782 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:39.130742    5315 generic.go:356] "Generic (PLEG): container finished" podID="c9d84bbc-3809-4942-9cf5-06038a7d0611" containerID="ccbaa2f218c71ebc1b53ec7a6860b9883d50f782338aa8f4697ce8e22baa80de" exitCode=0
Nov 03 18:28:39.131184 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:39.130796    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-58e97a-d97mn" event={"ID":"c9d84bbc-3809-4942-9cf5-06038a7d0611","Type":"ContainerDied","Data":"ccbaa2f218c71ebc1b53ec7a6860b9883d50f782338aa8f4697ce8e22baa80de"}
Nov 03 18:28:40.282997 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:40.282963    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-58e97a-d97mn"
Nov 03 18:28:40.349835 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:40.349794    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9d84bbc-3809-4942-9cf5-06038a7d0611-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9d84bbc-3809-4942-9cf5-06038a7d0611-maas-api-xks-ca\") pod \"c9d84bbc-3809-4942-9cf5-06038a7d0611\" (UID: \"c9d84bbc-3809-4942-9cf5-06038a7d0611\") "
Nov 03 18:28:40.350049 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:40.349922    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9d84bbc-3809-4942-9cf5-06038a7d0611-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9d84bbc-3809-4942-9cf5-06038a7d0611-maas-api-openshift-ca\") pod \"c9d84bbc-3809-4942-9cf5-06038a7d0611\" (UID: \"c9d84bbc-3809-4942-9cf5-06038a7d0611\") "
Nov 03 18:28:40.350049 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:40.350042    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9d84bbc-3809-4942-9cf5-06038a7d0611-kube-api-access-6mlkq\" (UniqueName: \"kubernetes.io/projected/c9d84bbc-3809-4942-9cf5-06038a7d0611-kube-api-access-6mlkq\") pod \"c9d84bbc-3809-4942-9cf5-06038a7d0611\" (UID: \"c9d84bbc-3809-4942-9cf5-06038a7d0611\") "
Nov 03 18:28:40.350294 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:40.350256    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9d84bbc-3809-4942-9cf5-06038a7d0611-maas-api-openshift-ca" pod "c9d84bbc-3809-4942-9cf5-06038a7d0611" (UID: "c9d84bbc-3809-4942-9cf5-06038a7d0611"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:28:40.352022 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:40.351993    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9d84bbc-3809-4942-9cf5-06038a7d0611-kube-api-access-6mlkq" pod "c9d84bbc-3809-4942-9cf5-06038a7d0611" (UID: "c9d84bbc-3809-4942-9cf5-06038a7d0611"). InnerVolumeSpecName "kube-api-access-6mlkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:28:40.352144 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:40.352074    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9d84bbc-3809-4942-9cf5-06038a7d0611-maas-api-xks-ca" pod "c9d84bbc-3809-4942-9cf5-06038a7d0611" (UID: "c9d84bbc-3809-4942-9cf5-06038a7d0611"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:28:40.451741 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:40.451653    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mlkq\" (UniqueName: \"kubernetes.io/projected/c9d84bbc-3809-4942-9cf5-06038a7d0611-kube-api-access-6mlkq\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:28:40.451741 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:40.451689    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9d84bbc-3809-4942-9cf5-06038a7d0611-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:28:40.451741 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:40.451699    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9d84bbc-3809-4942-9cf5-06038a7d0611-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:28:41.143468 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:41.143436    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-58e97a-d97mn"
Nov 03 18:28:41.143710 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:41.143411    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-58e97a-d97mn" event={"ID":"c9d84bbc-3809-4942-9cf5-06038a7d0611","Type":"ContainerDied","Data":"e96eb514d27273962d56c48f828280f82e0b741503001d29741cacdcdc67c013"}
Nov 03 18:28:41.143710 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:41.143547    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e96eb514d27273962d56c48f828280f82e0b741503001d29741cacdcdc67c013"
Nov 03 18:28:41.741325 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:41.741289    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-58e97a/simulator-single-main-58e97a-kserve-586d66fb65-svntp"]
Nov 03 18:28:54.578011 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:28:54.577962    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:28:55.045203 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:55.045171    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wqs8h"]
Nov 03 18:28:55.091291 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:55.091255    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wqs8h"]
Nov 03 18:28:55.102434 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:55.102400    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e656d6f-9d30-4600-b4ac-5fa8aa0cade2" path="/var/lib/kubelet/pods/2e656d6f-9d30-4600-b4ac-5fa8aa0cade2/volumes"
Nov 03 18:28:56.761936 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:56.761868    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-58e97a/simulator-single-main-58e97a-kserve-586d66fb65-svntp" podUID="21fb225c-08c4-443b-bde8-76b22a1a2aa0" containerName="main" containerID="cri-o://13c652dd6a0a39ba3b8bd9656cdfff6264235716d10acc58395dbd0de60b87c2" gracePeriod=45
Nov 03 18:28:56.867995 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:56.867954    5315 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-58e97a/simulator-single-main-58e97a-kserve-586d66fb65-svntp" podUID="21fb225c-08c4-443b-bde8-76b22a1a2aa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.243:8000/ready\": dial tcp 10.133.0.243:8000: connect: connection refused"
Nov 03 18:28:57.011478 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:57.011452    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-58e97a/simulator-single-main-58e97a-kserve-586d66fb65-svntp"
Nov 03 18:28:57.025597 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:57.025547    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21fb225c-08c4-443b-bde8-76b22a1a2aa0-home\" (UniqueName: \"kubernetes.io/empty-dir/21fb225c-08c4-443b-bde8-76b22a1a2aa0-home\") pod \"21fb225c-08c4-443b-bde8-76b22a1a2aa0\" (UID: \"21fb225c-08c4-443b-bde8-76b22a1a2aa0\") "
Nov 03 18:28:57.025700 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:57.025619    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21fb225c-08c4-443b-bde8-76b22a1a2aa0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/21fb225c-08c4-443b-bde8-76b22a1a2aa0-dshm\") pod \"21fb225c-08c4-443b-bde8-76b22a1a2aa0\" (UID: \"21fb225c-08c4-443b-bde8-76b22a1a2aa0\") "
Nov 03 18:28:57.025700 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:57.025679    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21fb225c-08c4-443b-bde8-76b22a1a2aa0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21fb225c-08c4-443b-bde8-76b22a1a2aa0-model-cache\") pod \"21fb225c-08c4-443b-bde8-76b22a1a2aa0\" (UID: \"21fb225c-08c4-443b-bde8-76b22a1a2aa0\") "
Nov 03 18:28:57.025784 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:57.025749    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21fb225c-08c4-443b-bde8-76b22a1a2aa0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21fb225c-08c4-443b-bde8-76b22a1a2aa0-tmp-dir\") pod \"21fb225c-08c4-443b-bde8-76b22a1a2aa0\" (UID: \"21fb225c-08c4-443b-bde8-76b22a1a2aa0\") "
Nov 03 18:28:57.025827 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:57.025791    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21fb225c-08c4-443b-bde8-76b22a1a2aa0-kube-api-access-jcljg\" (UniqueName: \"kubernetes.io/projected/21fb225c-08c4-443b-bde8-76b22a1a2aa0-kube-api-access-jcljg\") pod \"21fb225c-08c4-443b-bde8-76b22a1a2aa0\" (UID: \"21fb225c-08c4-443b-bde8-76b22a1a2aa0\") "
Nov 03 18:28:57.025827 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:57.025818    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21fb225c-08c4-443b-bde8-76b22a1a2aa0-tls-certs\" (UniqueName: \"kubernetes.io/secret/21fb225c-08c4-443b-bde8-76b22a1a2aa0-tls-certs\") pod \"21fb225c-08c4-443b-bde8-76b22a1a2aa0\" (UID: \"21fb225c-08c4-443b-bde8-76b22a1a2aa0\") "
Nov 03 18:28:57.025940 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:57.025851    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21fb225c-08c4-443b-bde8-76b22a1a2aa0-home" pod "21fb225c-08c4-443b-bde8-76b22a1a2aa0" (UID: "21fb225c-08c4-443b-bde8-76b22a1a2aa0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:28:57.026008 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:57.025966    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21fb225c-08c4-443b-bde8-76b22a1a2aa0-model-cache" pod "21fb225c-08c4-443b-bde8-76b22a1a2aa0" (UID: "21fb225c-08c4-443b-bde8-76b22a1a2aa0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:28:57.026134 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:57.026104    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21fb225c-08c4-443b-bde8-76b22a1a2aa0-tmp-dir" pod "21fb225c-08c4-443b-bde8-76b22a1a2aa0" (UID: "21fb225c-08c4-443b-bde8-76b22a1a2aa0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:28:57.026259 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:57.026236    5315 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21fb225c-08c4-443b-bde8-76b22a1a2aa0-model-cache\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:28:57.026259 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:57.026256    5315 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21fb225c-08c4-443b-bde8-76b22a1a2aa0-tmp-dir\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:28:57.026449 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:57.026266    5315 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21fb225c-08c4-443b-bde8-76b22a1a2aa0-home\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:28:57.027962 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:57.027936    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21fb225c-08c4-443b-bde8-76b22a1a2aa0-dshm" pod "21fb225c-08c4-443b-bde8-76b22a1a2aa0" (UID: "21fb225c-08c4-443b-bde8-76b22a1a2aa0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:28:57.028080 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:57.028060    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21fb225c-08c4-443b-bde8-76b22a1a2aa0-kube-api-access-jcljg" pod "21fb225c-08c4-443b-bde8-76b22a1a2aa0" (UID: "21fb225c-08c4-443b-bde8-76b22a1a2aa0"). InnerVolumeSpecName "kube-api-access-jcljg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:28:57.028125 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:57.028085    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21fb225c-08c4-443b-bde8-76b22a1a2aa0-tls-certs" pod "21fb225c-08c4-443b-bde8-76b22a1a2aa0" (UID: "21fb225c-08c4-443b-bde8-76b22a1a2aa0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:28:57.126803 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:57.126770    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcljg\" (UniqueName: \"kubernetes.io/projected/21fb225c-08c4-443b-bde8-76b22a1a2aa0-kube-api-access-jcljg\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:28:57.126803 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:57.126800    5315 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21fb225c-08c4-443b-bde8-76b22a1a2aa0-tls-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:28:57.127038 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:57.126812    5315 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21fb225c-08c4-443b-bde8-76b22a1a2aa0-dshm\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:28:57.247034 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:57.246992    5315 generic.go:356] "Generic (PLEG): container finished" podID="21fb225c-08c4-443b-bde8-76b22a1a2aa0" containerID="13c652dd6a0a39ba3b8bd9656cdfff6264235716d10acc58395dbd0de60b87c2" exitCode=0
Nov 03 18:28:57.247338 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:57.247311    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-58e97a/simulator-single-main-58e97a-kserve-586d66fb65-svntp" event={"ID":"21fb225c-08c4-443b-bde8-76b22a1a2aa0","Type":"ContainerDied","Data":"13c652dd6a0a39ba3b8bd9656cdfff6264235716d10acc58395dbd0de60b87c2"}
Nov 03 18:28:57.247770 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:57.247736    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-58e97a/simulator-single-main-58e97a-kserve-586d66fb65-svntp" event={"ID":"21fb225c-08c4-443b-bde8-76b22a1a2aa0","Type":"ContainerDied","Data":"03f69ee1bb1cc28e3924a80e4fcd64ee3bc0174caaba4bba6b75de0641d1eb46"}
Nov 03 18:28:57.247911 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:57.247897    5315 scope.go:122] "RemoveContainer" containerID="13c652dd6a0a39ba3b8bd9656cdfff6264235716d10acc58395dbd0de60b87c2"
Nov 03 18:28:57.248274 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:57.248255    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-58e97a/simulator-single-main-58e97a-kserve-586d66fb65-svntp"
Nov 03 18:28:57.273840 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:57.273814    5315 scope.go:122] "RemoveContainer" containerID="13c652dd6a0a39ba3b8bd9656cdfff6264235716d10acc58395dbd0de60b87c2"
Nov 03 18:28:57.274185 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:28:57.274157    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13c652dd6a0a39ba3b8bd9656cdfff6264235716d10acc58395dbd0de60b87c2\": container with ID starting with 13c652dd6a0a39ba3b8bd9656cdfff6264235716d10acc58395dbd0de60b87c2 not found: ID does not exist" containerID="13c652dd6a0a39ba3b8bd9656cdfff6264235716d10acc58395dbd0de60b87c2"
Nov 03 18:28:57.274282 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:57.274195    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13c652dd6a0a39ba3b8bd9656cdfff6264235716d10acc58395dbd0de60b87c2"} err="failed to get container status \"13c652dd6a0a39ba3b8bd9656cdfff6264235716d10acc58395dbd0de60b87c2\": rpc error: code = NotFound desc = could not find container \"13c652dd6a0a39ba3b8bd9656cdfff6264235716d10acc58395dbd0de60b87c2\": container with ID starting with 13c652dd6a0a39ba3b8bd9656cdfff6264235716d10acc58395dbd0de60b87c2 not found: ID does not exist"
Nov 03 18:28:57.403889 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:57.403857    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-58e97a/simulator-single-main-58e97a-kserve-586d66fb65-svntp"]
Nov 03 18:28:57.408052 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:57.408026    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-58e97a/simulator-single-main-58e97a-kserve-586d66fb65-svntp"]
Nov 03 18:28:57.454133 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:57.454099    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-58e97a-d97mn"]
Nov 03 18:28:57.456229 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:57.456206    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-58e97a-d97mn"]
Nov 03 18:28:59.103360 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:59.103321    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21fb225c-08c4-443b-bde8-76b22a1a2aa0" path="/var/lib/kubelet/pods/21fb225c-08c4-443b-bde8-76b22a1a2aa0/volumes"
Nov 03 18:28:59.103782 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:28:59.103748    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9d84bbc-3809-4942-9cf5-06038a7d0611" path="/var/lib/kubelet/pods/c9d84bbc-3809-4942-9cf5-06038a7d0611/volumes"
Nov 03 18:29:03.962166 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:03.962123    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj"]
Nov 03 18:29:03.962570 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:03.962404    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj" podUID="7a198348-f996-4bab-8499-21d41c5c9bee" containerName="istio-proxy" containerID="cri-o://c5000c8a4ad3b196d676fae0fb3df3ce4e822d52dba2a4a62db37a9055ad3ee7" gracePeriod=30
Nov 03 18:29:05.574859 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:05.574821    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-48fgm"]
Nov 03 18:29:05.592063 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:05.592030    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-48fgm"]
Nov 03 18:29:05.700494 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:05.700457    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zg4ns"]
Nov 03 18:29:05.700998 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:05.700975    5315 cpu_manager.go:414] "containerMap: removing container" podUID="21fb225c-08c4-443b-bde8-76b22a1a2aa0" containerName="main"
Nov 03 18:29:05.701120 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:05.700999    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21fb225c-08c4-443b-bde8-76b22a1a2aa0" containerName="main"
Nov 03 18:29:05.701120 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:05.701027    5315 cpu_manager.go:414] "containerMap: removing container" podUID="c9d84bbc-3809-4942-9cf5-06038a7d0611" containerName="revoke-keys"
Nov 03 18:29:05.701120 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:05.701039    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9d84bbc-3809-4942-9cf5-06038a7d0611" containerName="revoke-keys"
Nov 03 18:29:05.701237 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:05.701135    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="21fb225c-08c4-443b-bde8-76b22a1a2aa0" containerName="main"
Nov 03 18:29:05.701237 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:05.701145    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9d84bbc-3809-4942-9cf5-06038a7d0611" containerName="revoke-keys"
Nov 03 18:29:05.705545 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:05.705342    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zg4ns"
Nov 03 18:29:05.754349 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:05.754315    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:29:05.754530 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:05.754509    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:29:05.754603 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:05.754529    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:29:05.754603 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:05.754575    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:29:05.764057 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:05.764031    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zg4ns"]
Nov 03 18:29:05.813235 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:05.813197    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mj5g\" (UniqueName: \"kubernetes.io/projected/3961efda-ba14-40d9-88c4-74d4d4882900-kube-api-access-7mj5g\") pod \"maas-api-revoke-sub-3d4156f1-zg4ns\" (UID: \"3961efda-ba14-40d9-88c4-74d4d4882900\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zg4ns"
Nov 03 18:29:05.813445 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:05.813251    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3961efda-ba14-40d9-88c4-74d4d4882900-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-zg4ns\" (UID: \"3961efda-ba14-40d9-88c4-74d4d4882900\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zg4ns"
Nov 03 18:29:05.813445 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:05.813351    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3961efda-ba14-40d9-88c4-74d4d4882900-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-zg4ns\" (UID: \"3961efda-ba14-40d9-88c4-74d4d4882900\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zg4ns"
Nov 03 18:29:05.914053 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:05.913954    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3961efda-ba14-40d9-88c4-74d4d4882900-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-zg4ns\" (UID: \"3961efda-ba14-40d9-88c4-74d4d4882900\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zg4ns"
Nov 03 18:29:05.914053 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:05.914019    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3961efda-ba14-40d9-88c4-74d4d4882900-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-zg4ns\" (UID: \"3961efda-ba14-40d9-88c4-74d4d4882900\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zg4ns"
Nov 03 18:29:05.914310 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:05.914146    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mj5g\" (UniqueName: \"kubernetes.io/projected/3961efda-ba14-40d9-88c4-74d4d4882900-kube-api-access-7mj5g\") pod \"maas-api-revoke-sub-3d4156f1-zg4ns\" (UID: \"3961efda-ba14-40d9-88c4-74d4d4882900\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zg4ns"
Nov 03 18:29:05.914828 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:05.914800    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3961efda-ba14-40d9-88c4-74d4d4882900-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-zg4ns\" (UID: \"3961efda-ba14-40d9-88c4-74d4d4882900\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zg4ns"
Nov 03 18:29:05.916489 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:05.916464    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3961efda-ba14-40d9-88c4-74d4d4882900-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-zg4ns\" (UID: \"3961efda-ba14-40d9-88c4-74d4d4882900\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zg4ns"
Nov 03 18:29:05.971671 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:05.971635    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mj5g\" (UniqueName: \"kubernetes.io/projected/3961efda-ba14-40d9-88c4-74d4d4882900-kube-api-access-7mj5g\") pod \"maas-api-revoke-sub-3d4156f1-zg4ns\" (UID: \"3961efda-ba14-40d9-88c4-74d4d4882900\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zg4ns"
Nov 03 18:29:05.995834 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:05.995796    5315 scope.go:122] "RemoveContainer" containerID="8d4ded2b6669cd2e9d6336156a226f44b0038c14a649d7a87638238842721503"
Nov 03 18:29:06.004232 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:06.004212    5315 scope.go:122] "RemoveContainer" containerID="d36d7f02c8fecfbbc3e4403aff9882aa6a215cfea8fc6b3edd3bc6f5f794d124"
Nov 03 18:29:06.018014 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:06.017979    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zg4ns"
Nov 03 18:29:06.388670 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:06.388636    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zg4ns"]
Nov 03 18:29:06.395781 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:29:06.395740    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3961efda_ba14_40d9_88c4_74d4d4882900.slice/crio-2d8d7c29c5e80c2dcbc28838ebd7501cfee36abec5aee0a4b6da5d7f25136989 WatchSource:0}: Error finding container 2d8d7c29c5e80c2dcbc28838ebd7501cfee36abec5aee0a4b6da5d7f25136989: Status 404 returned error can't find the container with id 2d8d7c29c5e80c2dcbc28838ebd7501cfee36abec5aee0a4b6da5d7f25136989
Nov 03 18:29:07.102693 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:07.102658    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85849579-f963-4e58-b330-eb25238efaec" path="/var/lib/kubelet/pods/85849579-f963-4e58-b330-eb25238efaec/volumes"
Nov 03 18:29:07.297812 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:07.297769    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zg4ns" event={"ID":"3961efda-ba14-40d9-88c4-74d4d4882900","Type":"ContainerStarted","Data":"08f6cf20e01c2687d6bf7cfc1aef9a1490577bbcfe70e6960ea0d5d999d3e629"}
Nov 03 18:29:07.297998 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:07.297846    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zg4ns" event={"ID":"3961efda-ba14-40d9-88c4-74d4d4882900","Type":"ContainerStarted","Data":"2d8d7c29c5e80c2dcbc28838ebd7501cfee36abec5aee0a4b6da5d7f25136989"}
Nov 03 18:29:07.454295 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:07.454178    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zg4ns" podStartSLOduration=2.454163064 podStartE2EDuration="2.454163064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:29:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:29:07.407237654 +0000 UTC m=+2586.423435724" watchObservedRunningTime="2026-11-03 18:29:07.454163064 +0000 UTC m=+2586.470361127"
Nov 03 18:29:08.303753 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:08.303712    5315 generic.go:356] "Generic (PLEG): container finished" podID="3961efda-ba14-40d9-88c4-74d4d4882900" containerID="08f6cf20e01c2687d6bf7cfc1aef9a1490577bbcfe70e6960ea0d5d999d3e629" exitCode=0
Nov 03 18:29:08.304178 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:08.303801    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zg4ns" event={"ID":"3961efda-ba14-40d9-88c4-74d4d4882900","Type":"ContainerDied","Data":"08f6cf20e01c2687d6bf7cfc1aef9a1490577bbcfe70e6960ea0d5d999d3e629"}
Nov 03 18:29:09.218790 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:09.218763    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj"
Nov 03 18:29:09.310337 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:09.310305    5315 generic.go:356] "Generic (PLEG): container finished" podID="7a198348-f996-4bab-8499-21d41c5c9bee" containerID="c5000c8a4ad3b196d676fae0fb3df3ce4e822d52dba2a4a62db37a9055ad3ee7" exitCode=0
Nov 03 18:29:09.310776 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:09.310356    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj"
Nov 03 18:29:09.310776 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:09.310395    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj" event={"ID":"7a198348-f996-4bab-8499-21d41c5c9bee","Type":"ContainerDied","Data":"c5000c8a4ad3b196d676fae0fb3df3ce4e822d52dba2a4a62db37a9055ad3ee7"}
Nov 03 18:29:09.310776 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:09.310446    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj" event={"ID":"7a198348-f996-4bab-8499-21d41c5c9bee","Type":"ContainerDied","Data":"776214ce4bc652330c9845901f603a332e2e41497da088d9d2be01298bc68d1a"}
Nov 03 18:29:09.310776 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:09.310475    5315 scope.go:122] "RemoveContainer" containerID="c5000c8a4ad3b196d676fae0fb3df3ce4e822d52dba2a4a62db37a9055ad3ee7"
Nov 03 18:29:09.319389 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:09.319365    5315 scope.go:122] "RemoveContainer" containerID="75019fa6048002a549cc9c9b6e0ea4f03e673173285b36f22d5cdba173a5dccb"
Nov 03 18:29:09.329705 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:09.329681    5315 scope.go:122] "RemoveContainer" containerID="c5000c8a4ad3b196d676fae0fb3df3ce4e822d52dba2a4a62db37a9055ad3ee7"
Nov 03 18:29:09.330022 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:29:09.330002    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5000c8a4ad3b196d676fae0fb3df3ce4e822d52dba2a4a62db37a9055ad3ee7\": container with ID starting with c5000c8a4ad3b196d676fae0fb3df3ce4e822d52dba2a4a62db37a9055ad3ee7 not found: ID does not exist" containerID="c5000c8a4ad3b196d676fae0fb3df3ce4e822d52dba2a4a62db37a9055ad3ee7"
Nov 03 18:29:09.330088 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:09.330035    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5000c8a4ad3b196d676fae0fb3df3ce4e822d52dba2a4a62db37a9055ad3ee7"} err="failed to get container status \"c5000c8a4ad3b196d676fae0fb3df3ce4e822d52dba2a4a62db37a9055ad3ee7\": rpc error: code = NotFound desc = could not find container \"c5000c8a4ad3b196d676fae0fb3df3ce4e822d52dba2a4a62db37a9055ad3ee7\": container with ID starting with c5000c8a4ad3b196d676fae0fb3df3ce4e822d52dba2a4a62db37a9055ad3ee7 not found: ID does not exist"
Nov 03 18:29:09.330088 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:09.330055    5315 scope.go:122] "RemoveContainer" containerID="75019fa6048002a549cc9c9b6e0ea4f03e673173285b36f22d5cdba173a5dccb"
Nov 03 18:29:09.330337 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:29:09.330309    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75019fa6048002a549cc9c9b6e0ea4f03e673173285b36f22d5cdba173a5dccb\": container with ID starting with 75019fa6048002a549cc9c9b6e0ea4f03e673173285b36f22d5cdba173a5dccb not found: ID does not exist" containerID="75019fa6048002a549cc9c9b6e0ea4f03e673173285b36f22d5cdba173a5dccb"
Nov 03 18:29:09.330400 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:09.330341    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75019fa6048002a549cc9c9b6e0ea4f03e673173285b36f22d5cdba173a5dccb"} err="failed to get container status \"75019fa6048002a549cc9c9b6e0ea4f03e673173285b36f22d5cdba173a5dccb\": rpc error: code = NotFound desc = could not find container \"75019fa6048002a549cc9c9b6e0ea4f03e673173285b36f22d5cdba173a5dccb\": container with ID starting with 75019fa6048002a549cc9c9b6e0ea4f03e673173285b36f22d5cdba173a5dccb not found: ID does not exist"
Nov 03 18:29:09.349461 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:09.349428    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a198348-f996-4bab-8499-21d41c5c9bee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a198348-f996-4bab-8499-21d41c5c9bee-workload-socket\") pod \"7a198348-f996-4bab-8499-21d41c5c9bee\" (UID: \"7a198348-f996-4bab-8499-21d41c5c9bee\") "
Nov 03 18:29:09.349646 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:09.349474    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a198348-f996-4bab-8499-21d41c5c9bee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a198348-f996-4bab-8499-21d41c5c9bee-istio-data\") pod \"7a198348-f996-4bab-8499-21d41c5c9bee\" (UID: \"7a198348-f996-4bab-8499-21d41c5c9bee\") "
Nov 03 18:29:09.349646 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:09.349516    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a198348-f996-4bab-8499-21d41c5c9bee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a198348-f996-4bab-8499-21d41c5c9bee-istio-envoy\") pod \"7a198348-f996-4bab-8499-21d41c5c9bee\" (UID: \"7a198348-f996-4bab-8499-21d41c5c9bee\") "
Nov 03 18:29:09.349646 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:09.349548    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a198348-f996-4bab-8499-21d41c5c9bee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a198348-f996-4bab-8499-21d41c5c9bee-istio-podinfo\") pod \"7a198348-f996-4bab-8499-21d41c5c9bee\" (UID: \"7a198348-f996-4bab-8499-21d41c5c9bee\") "
Nov 03 18:29:09.349646 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:09.349603    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a198348-f996-4bab-8499-21d41c5c9bee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a198348-f996-4bab-8499-21d41c5c9bee-credential-socket\") pod \"7a198348-f996-4bab-8499-21d41c5c9bee\" (UID: \"7a198348-f996-4bab-8499-21d41c5c9bee\") "
Nov 03 18:29:09.349877 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:09.349679    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a198348-f996-4bab-8499-21d41c5c9bee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a198348-f996-4bab-8499-21d41c5c9bee-workload-certs\") pod \"7a198348-f996-4bab-8499-21d41c5c9bee\" (UID: \"7a198348-f996-4bab-8499-21d41c5c9bee\") "
Nov 03 18:29:09.349877 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:09.349751    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a198348-f996-4bab-8499-21d41c5c9bee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a198348-f996-4bab-8499-21d41c5c9bee-istiod-ca-cert\") pod \"7a198348-f996-4bab-8499-21d41c5c9bee\" (UID: \"7a198348-f996-4bab-8499-21d41c5c9bee\") "
Nov 03 18:29:09.349877 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:09.349767    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a198348-f996-4bab-8499-21d41c5c9bee-workload-socket" pod "7a198348-f996-4bab-8499-21d41c5c9bee" (UID: "7a198348-f996-4bab-8499-21d41c5c9bee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:29:09.349877 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:09.349783    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a198348-f996-4bab-8499-21d41c5c9bee-istio-data" pod "7a198348-f996-4bab-8499-21d41c5c9bee" (UID: "7a198348-f996-4bab-8499-21d41c5c9bee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:29:09.349877 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:09.349793    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a198348-f996-4bab-8499-21d41c5c9bee-istio-token\" (UniqueName: \"kubernetes.io/projected/7a198348-f996-4bab-8499-21d41c5c9bee-istio-token\") pod \"7a198348-f996-4bab-8499-21d41c5c9bee\" (UID: \"7a198348-f996-4bab-8499-21d41c5c9bee\") "
Nov 03 18:29:09.349877 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:09.349852    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a198348-f996-4bab-8499-21d41c5c9bee-kube-api-access-2q968\" (UniqueName: \"kubernetes.io/projected/7a198348-f996-4bab-8499-21d41c5c9bee-kube-api-access-2q968\") pod \"7a198348-f996-4bab-8499-21d41c5c9bee\" (UID: \"7a198348-f996-4bab-8499-21d41c5c9bee\") "
Nov 03 18:29:09.350217 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:09.350127    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a198348-f996-4bab-8499-21d41c5c9bee-credential-socket" pod "7a198348-f996-4bab-8499-21d41c5c9bee" (UID: "7a198348-f996-4bab-8499-21d41c5c9bee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:29:09.350285 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:09.350263    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a198348-f996-4bab-8499-21d41c5c9bee-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:29:09.350345 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:09.350283    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a198348-f996-4bab-8499-21d41c5c9bee-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:29:09.350345 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:09.350295    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a198348-f996-4bab-8499-21d41c5c9bee-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:29:09.350345 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:09.350290    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a198348-f996-4bab-8499-21d41c5c9bee-istiod-ca-cert" pod "7a198348-f996-4bab-8499-21d41c5c9bee" (UID: "7a198348-f996-4bab-8499-21d41c5c9bee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:29:09.350513 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:09.350351    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a198348-f996-4bab-8499-21d41c5c9bee-workload-certs" pod "7a198348-f996-4bab-8499-21d41c5c9bee" (UID: "7a198348-f996-4bab-8499-21d41c5c9bee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:29:09.351947 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:09.351917    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a198348-f996-4bab-8499-21d41c5c9bee-istio-podinfo" pod "7a198348-f996-4bab-8499-21d41c5c9bee" (UID: "7a198348-f996-4bab-8499-21d41c5c9bee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:29:09.352094 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:09.352006    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a198348-f996-4bab-8499-21d41c5c9bee-istio-token" pod "7a198348-f996-4bab-8499-21d41c5c9bee" (UID: "7a198348-f996-4bab-8499-21d41c5c9bee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:29:09.352164 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:09.352091    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a198348-f996-4bab-8499-21d41c5c9bee-kube-api-access-2q968" pod "7a198348-f996-4bab-8499-21d41c5c9bee" (UID: "7a198348-f996-4bab-8499-21d41c5c9bee"). InnerVolumeSpecName "kube-api-access-2q968". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:29:09.352401 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:09.352386    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a198348-f996-4bab-8499-21d41c5c9bee-istio-envoy" pod "7a198348-f996-4bab-8499-21d41c5c9bee" (UID: "7a198348-f996-4bab-8499-21d41c5c9bee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:29:09.433302 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:09.433276    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zg4ns"
Nov 03 18:29:09.451217 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:09.451186    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a198348-f996-4bab-8499-21d41c5c9bee-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:29:09.451217 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:09.451216    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a198348-f996-4bab-8499-21d41c5c9bee-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:29:09.451404 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:09.451230    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a198348-f996-4bab-8499-21d41c5c9bee-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:29:09.451404 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:09.451242    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q968\" (UniqueName: \"kubernetes.io/projected/7a198348-f996-4bab-8499-21d41c5c9bee-kube-api-access-2q968\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:29:09.451404 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:09.451257    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a198348-f996-4bab-8499-21d41c5c9bee-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:29:09.451404 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:09.451268    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a198348-f996-4bab-8499-21d41c5c9bee-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:29:09.552531 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:09.552497    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3961efda-ba14-40d9-88c4-74d4d4882900-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3961efda-ba14-40d9-88c4-74d4d4882900-maas-api-openshift-ca\") pod \"3961efda-ba14-40d9-88c4-74d4d4882900\" (UID: \"3961efda-ba14-40d9-88c4-74d4d4882900\") "
Nov 03 18:29:09.552720 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:09.552599    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3961efda-ba14-40d9-88c4-74d4d4882900-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3961efda-ba14-40d9-88c4-74d4d4882900-maas-api-xks-ca\") pod \"3961efda-ba14-40d9-88c4-74d4d4882900\" (UID: \"3961efda-ba14-40d9-88c4-74d4d4882900\") "
Nov 03 18:29:09.552720 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:09.552652    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3961efda-ba14-40d9-88c4-74d4d4882900-kube-api-access-7mj5g\" (UniqueName: \"kubernetes.io/projected/3961efda-ba14-40d9-88c4-74d4d4882900-kube-api-access-7mj5g\") pod \"3961efda-ba14-40d9-88c4-74d4d4882900\" (UID: \"3961efda-ba14-40d9-88c4-74d4d4882900\") "
Nov 03 18:29:09.552909 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:09.552883    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3961efda-ba14-40d9-88c4-74d4d4882900-maas-api-openshift-ca" pod "3961efda-ba14-40d9-88c4-74d4d4882900" (UID: "3961efda-ba14-40d9-88c4-74d4d4882900"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:29:09.553020 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:09.553002    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3961efda-ba14-40d9-88c4-74d4d4882900-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:29:09.554828 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:09.554799    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3961efda-ba14-40d9-88c4-74d4d4882900-kube-api-access-7mj5g" pod "3961efda-ba14-40d9-88c4-74d4d4882900" (UID: "3961efda-ba14-40d9-88c4-74d4d4882900"). InnerVolumeSpecName "kube-api-access-7mj5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:29:09.555131 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:09.555108    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3961efda-ba14-40d9-88c4-74d4d4882900-maas-api-xks-ca" pod "3961efda-ba14-40d9-88c4-74d4d4882900" (UID: "3961efda-ba14-40d9-88c4-74d4d4882900"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:29:09.654248 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:09.654206    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3961efda-ba14-40d9-88c4-74d4d4882900-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:29:09.654248 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:09.654239    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mj5g\" (UniqueName: \"kubernetes.io/projected/3961efda-ba14-40d9-88c4-74d4d4882900-kube-api-access-7mj5g\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:29:09.722988 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:09.722951    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj"]
Nov 03 18:29:09.727309 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:09.727276    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-58e97a-openshift-default-f87f6fcdf-gpswj"]
Nov 03 18:29:10.318131 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:10.318104    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zg4ns"
Nov 03 18:29:10.318571 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:10.318125    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zg4ns" event={"ID":"3961efda-ba14-40d9-88c4-74d4d4882900","Type":"ContainerDied","Data":"2d8d7c29c5e80c2dcbc28838ebd7501cfee36abec5aee0a4b6da5d7f25136989"}
Nov 03 18:29:10.318571 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:10.318170    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d8d7c29c5e80c2dcbc28838ebd7501cfee36abec5aee0a4b6da5d7f25136989"
Nov 03 18:29:11.105836 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:11.103385    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a198348-f996-4bab-8499-21d41c5c9bee" path="/var/lib/kubelet/pods/7a198348-f996-4bab-8499-21d41c5c9bee/volumes"
Nov 03 18:29:23.044594 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:23.044558    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fr4v5"]
Nov 03 18:29:23.090762 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:23.090723    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fr4v5"]
Nov 03 18:29:23.102474 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:23.102440    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7d79930-2bdb-46ed-8f43-2722e34f8948" path="/var/lib/kubelet/pods/a7d79930-2bdb-46ed-8f43-2722e34f8948/volumes"
Nov 03 18:29:24.577030 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:29:24.576983    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:29:54.045115 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:54.045075    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vrlz9"]
Nov 03 18:29:54.091117 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:54.091083    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vrlz9"]
Nov 03 18:29:54.577780 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:29:54.577739    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:29:55.102193 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:29:55.102150    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0196849-40f8-4d47-a66d-673b84140384" path="/var/lib/kubelet/pods/c0196849-40f8-4d47-a66d-673b84140384/volumes"
Nov 03 18:30:00.211695 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:00.211655    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895510-bxpbt"]
Nov 03 18:30:00.212105 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:00.212027    5315 cpu_manager.go:414] "containerMap: removing container" podUID="3961efda-ba14-40d9-88c4-74d4d4882900" containerName="revoke-keys"
Nov 03 18:30:00.212105 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:00.212038    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3961efda-ba14-40d9-88c4-74d4d4882900" containerName="revoke-keys"
Nov 03 18:30:00.212105 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:00.212069    5315 cpu_manager.go:414] "containerMap: removing container" podUID="7a198348-f996-4bab-8499-21d41c5c9bee" containerName="istio-proxy"
Nov 03 18:30:00.212105 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:00.212075    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a198348-f996-4bab-8499-21d41c5c9bee" containerName="istio-proxy"
Nov 03 18:30:00.212105 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:00.212089    5315 cpu_manager.go:414] "containerMap: removing container" podUID="7a198348-f996-4bab-8499-21d41c5c9bee" containerName="istio-proxy"
Nov 03 18:30:00.212105 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:00.212094    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a198348-f996-4bab-8499-21d41c5c9bee" containerName="istio-proxy"
Nov 03 18:30:00.212316 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:00.212163    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a198348-f996-4bab-8499-21d41c5c9bee" containerName="istio-proxy"
Nov 03 18:30:00.212316 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:00.212172    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a198348-f996-4bab-8499-21d41c5c9bee" containerName="istio-proxy"
Nov 03 18:30:00.212316 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:00.212182    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="3961efda-ba14-40d9-88c4-74d4d4882900" containerName="revoke-keys"
Nov 03 18:30:00.215238 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:00.215215    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-bxpbt"
Nov 03 18:30:00.263473 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:00.263437    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:30:00.263697 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:00.263497    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:30:00.309758 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:00.309722    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:30:00.311110 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:00.311084    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895510-bxpbt"]
Nov 03 18:30:00.334124 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:00.334089    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad0aee0e-6395-4067-a9da-9a5055c711cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895510-bxpbt\" (UID: \"ad0aee0e-6395-4067-a9da-9a5055c711cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-bxpbt"
Nov 03 18:30:00.334280 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:00.334131    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwwxj\" (UniqueName: \"kubernetes.io/projected/ad0aee0e-6395-4067-a9da-9a5055c711cc-kube-api-access-vwwxj\") pod \"maas-api-key-cleanup-29895510-bxpbt\" (UID: \"ad0aee0e-6395-4067-a9da-9a5055c711cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-bxpbt"
Nov 03 18:30:00.435702 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:00.435662    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad0aee0e-6395-4067-a9da-9a5055c711cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895510-bxpbt\" (UID: \"ad0aee0e-6395-4067-a9da-9a5055c711cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-bxpbt"
Nov 03 18:30:00.435702 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:00.435704    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwwxj\" (UniqueName: \"kubernetes.io/projected/ad0aee0e-6395-4067-a9da-9a5055c711cc-kube-api-access-vwwxj\") pod \"maas-api-key-cleanup-29895510-bxpbt\" (UID: \"ad0aee0e-6395-4067-a9da-9a5055c711cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-bxpbt"
Nov 03 18:30:00.436295 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:00.436271    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad0aee0e-6395-4067-a9da-9a5055c711cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895510-bxpbt\" (UID: \"ad0aee0e-6395-4067-a9da-9a5055c711cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-bxpbt"
Nov 03 18:30:00.493210 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:00.493138    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwwxj\" (UniqueName: \"kubernetes.io/projected/ad0aee0e-6395-4067-a9da-9a5055c711cc-kube-api-access-vwwxj\") pod \"maas-api-key-cleanup-29895510-bxpbt\" (UID: \"ad0aee0e-6395-4067-a9da-9a5055c711cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-bxpbt"
Nov 03 18:30:00.527918 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:00.527887    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-bxpbt"
Nov 03 18:30:00.899038 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:00.899007    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895510-bxpbt"]
Nov 03 18:30:00.906301 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:30:00.906270    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad0aee0e_6395_4067_a9da_9a5055c711cc.slice/crio-130bc13cb1a8f14da9501a995e3523398129b494c7f7a28034f59f3bdc688dab WatchSource:0}: Error finding container 130bc13cb1a8f14da9501a995e3523398129b494c7f7a28034f59f3bdc688dab: Status 404 returned error can't find the container with id 130bc13cb1a8f14da9501a995e3523398129b494c7f7a28034f59f3bdc688dab
Nov 03 18:30:01.570448 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:01.570404    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-bxpbt" event={"ID":"ad0aee0e-6395-4067-a9da-9a5055c711cc","Type":"ContainerStarted","Data":"02c92cf95a884ac2704712419180e374ed9bd92076291cd590113f23ada27908"}
Nov 03 18:30:01.570989 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:01.570470    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-bxpbt" event={"ID":"ad0aee0e-6395-4067-a9da-9a5055c711cc","Type":"ContainerStarted","Data":"130bc13cb1a8f14da9501a995e3523398129b494c7f7a28034f59f3bdc688dab"}
Nov 03 18:30:01.679330 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:01.679268    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-bxpbt" podStartSLOduration=1.679252951 podStartE2EDuration="1.679252951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:30:01.67797883 +0000 UTC m=+2640.694176899" watchObservedRunningTime="2026-11-03 18:30:01.679252951 +0000 UTC m=+2640.695451041"
Nov 03 18:30:02.578824 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:02.578777    5315 generic.go:356] "Generic (PLEG): container finished" podID="ad0aee0e-6395-4067-a9da-9a5055c711cc" containerID="02c92cf95a884ac2704712419180e374ed9bd92076291cd590113f23ada27908" exitCode=60
Nov 03 18:30:02.579337 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:02.578927    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-bxpbt" event={"ID":"ad0aee0e-6395-4067-a9da-9a5055c711cc","Type":"ContainerDied","Data":"02c92cf95a884ac2704712419180e374ed9bd92076291cd590113f23ada27908"}
Nov 03 18:30:02.579451 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:02.579427    5315 scope.go:122] "RemoveContainer" containerID="02c92cf95a884ac2704712419180e374ed9bd92076291cd590113f23ada27908"
Nov 03 18:30:03.585044 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:03.584959    5315 generic.go:356] "Generic (PLEG): container finished" podID="ad0aee0e-6395-4067-a9da-9a5055c711cc" containerID="f4bb1bb118f1d7d8360268642fc47b915c63e77c5b9f083d23b32da713b9f779" exitCode=60
Nov 03 18:30:03.585480 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:03.585046    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-bxpbt" event={"ID":"ad0aee0e-6395-4067-a9da-9a5055c711cc","Type":"ContainerDied","Data":"f4bb1bb118f1d7d8360268642fc47b915c63e77c5b9f083d23b32da713b9f779"}
Nov 03 18:30:03.585480 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:03.585104    5315 scope.go:122] "RemoveContainer" containerID="02c92cf95a884ac2704712419180e374ed9bd92076291cd590113f23ada27908"
Nov 03 18:30:03.585480 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:03.585357    5315 scope.go:122] "RemoveContainer" containerID="f4bb1bb118f1d7d8360268642fc47b915c63e77c5b9f083d23b32da713b9f779"
Nov 03 18:30:03.585640 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:30:03.585591    5315 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895510-bxpbt_odh-ai-gateway-infra(ad0aee0e-6395-4067-a9da-9a5055c711cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-bxpbt" podUID="ad0aee0e-6395-4067-a9da-9a5055c711cc"
Nov 03 18:30:04.591067 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:04.591040    5315 scope.go:122] "RemoveContainer" containerID="f4bb1bb118f1d7d8360268642fc47b915c63e77c5b9f083d23b32da713b9f779"
Nov 03 18:30:04.591475 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:30:04.591250    5315 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895510-bxpbt_odh-ai-gateway-infra(ad0aee0e-6395-4067-a9da-9a5055c711cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-bxpbt" podUID="ad0aee0e-6395-4067-a9da-9a5055c711cc"
Nov 03 18:30:06.035594 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:06.035555    5315 scope.go:122] "RemoveContainer" containerID="35b2bc8e1b347aa6d78b80e101e5e0553e62905df07f7e9cfeec705b7c6ec4d4"
Nov 03 18:30:06.044069 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:06.044047    5315 scope.go:122] "RemoveContainer" containerID="ea6f8e047e20ac33836dcbf00a00460d1cfe6b5808bfabe0464a8f8f2726c82a"
Nov 03 18:30:06.051683 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:06.051654    5315 scope.go:122] "RemoveContainer" containerID="9603f8f33344f6369e614279a3734dfe216411fc716fa4150b96c77310251897"
Nov 03 18:30:14.097161 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:14.097123    5315 scope.go:122] "RemoveContainer" containerID="f4bb1bb118f1d7d8360268642fc47b915c63e77c5b9f083d23b32da713b9f779"
Nov 03 18:30:14.639957 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:14.639920    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-bxpbt" event={"ID":"ad0aee0e-6395-4067-a9da-9a5055c711cc","Type":"ContainerStarted","Data":"e52b609c4a61f1677e06ab693456d3e6f2a5deb1aa536e8ecc1c722397399a3c"}
Nov 03 18:30:15.046492 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:15.046459    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cslvm"]
Nov 03 18:30:15.048287 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:15.048257    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cslvm"]
Nov 03 18:30:15.102848 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:15.102760    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8673571d-193d-4120-b63a-e517f961d06f" path="/var/lib/kubelet/pods/8673571d-193d-4120-b63a-e517f961d06f/volumes"
Nov 03 18:30:15.646670 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:15.646633    5315 generic.go:356] "Generic (PLEG): container finished" podID="ad0aee0e-6395-4067-a9da-9a5055c711cc" containerID="e52b609c4a61f1677e06ab693456d3e6f2a5deb1aa536e8ecc1c722397399a3c" exitCode=60
Nov 03 18:30:15.646834 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:15.646687    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-bxpbt" event={"ID":"ad0aee0e-6395-4067-a9da-9a5055c711cc","Type":"ContainerDied","Data":"e52b609c4a61f1677e06ab693456d3e6f2a5deb1aa536e8ecc1c722397399a3c"}
Nov 03 18:30:15.646834 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:15.646769    5315 scope.go:122] "RemoveContainer" containerID="f4bb1bb118f1d7d8360268642fc47b915c63e77c5b9f083d23b32da713b9f779"
Nov 03 18:30:15.647160 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:15.647137    5315 scope.go:122] "RemoveContainer" containerID="e52b609c4a61f1677e06ab693456d3e6f2a5deb1aa536e8ecc1c722397399a3c"
Nov 03 18:30:15.647406 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:30:15.647386    5315 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895510-bxpbt_odh-ai-gateway-infra(ad0aee0e-6395-4067-a9da-9a5055c711cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-bxpbt" podUID="ad0aee0e-6395-4067-a9da-9a5055c711cc"
Nov 03 18:30:15.762131 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:15.762096    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895510-bxpbt"]
Nov 03 18:30:16.780650 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:16.780601    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-bxpbt"
Nov 03 18:30:16.889323 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:16.889276    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad0aee0e-6395-4067-a9da-9a5055c711cc-kube-api-access-vwwxj\" (UniqueName: \"kubernetes.io/projected/ad0aee0e-6395-4067-a9da-9a5055c711cc-kube-api-access-vwwxj\") pod \"ad0aee0e-6395-4067-a9da-9a5055c711cc\" (UID: \"ad0aee0e-6395-4067-a9da-9a5055c711cc\") "
Nov 03 18:30:16.889530 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:16.889449    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad0aee0e-6395-4067-a9da-9a5055c711cc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad0aee0e-6395-4067-a9da-9a5055c711cc-maas-api-service-ca\") pod \"ad0aee0e-6395-4067-a9da-9a5055c711cc\" (UID: \"ad0aee0e-6395-4067-a9da-9a5055c711cc\") "
Nov 03 18:30:16.889840 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:16.889809    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad0aee0e-6395-4067-a9da-9a5055c711cc-maas-api-service-ca" pod "ad0aee0e-6395-4067-a9da-9a5055c711cc" (UID: "ad0aee0e-6395-4067-a9da-9a5055c711cc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:30:16.891480 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:16.891450    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad0aee0e-6395-4067-a9da-9a5055c711cc-kube-api-access-vwwxj" pod "ad0aee0e-6395-4067-a9da-9a5055c711cc" (UID: "ad0aee0e-6395-4067-a9da-9a5055c711cc"). InnerVolumeSpecName "kube-api-access-vwwxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:30:16.990902 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:16.990811    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad0aee0e-6395-4067-a9da-9a5055c711cc-maas-api-service-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:30:16.990902 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:16.990842    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwwxj\" (UniqueName: \"kubernetes.io/projected/ad0aee0e-6395-4067-a9da-9a5055c711cc-kube-api-access-vwwxj\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:30:17.658872 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:17.658839    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-bxpbt"
Nov 03 18:30:17.659092 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:17.658842    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-bxpbt" event={"ID":"ad0aee0e-6395-4067-a9da-9a5055c711cc","Type":"ContainerDied","Data":"130bc13cb1a8f14da9501a995e3523398129b494c7f7a28034f59f3bdc688dab"}
Nov 03 18:30:17.659092 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:17.658977    5315 scope.go:122] "RemoveContainer" containerID="e52b609c4a61f1677e06ab693456d3e6f2a5deb1aa536e8ecc1c722397399a3c"
Nov 03 18:30:17.810208 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:17.810172    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895510-bxpbt"]
Nov 03 18:30:17.856294 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:17.856255    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895510-bxpbt"]
Nov 03 18:30:19.102158 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:19.102122    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad0aee0e-6395-4067-a9da-9a5055c711cc" path="/var/lib/kubelet/pods/ad0aee0e-6395-4067-a9da-9a5055c711cc/volumes"
Nov 03 18:30:24.576370 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:30:24.576332    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:30:54.576708 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:30:54.576666    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:30:58.859346 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:58.859309    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-lm6tb"]
Nov 03 18:30:58.859889 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:58.859686    5315 cpu_manager.go:414] "containerMap: removing container" podUID="ad0aee0e-6395-4067-a9da-9a5055c711cc" containerName="cleanup"
Nov 03 18:30:58.859889 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:58.859699    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad0aee0e-6395-4067-a9da-9a5055c711cc" containerName="cleanup"
Nov 03 18:30:58.859889 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:58.859712    5315 cpu_manager.go:414] "containerMap: removing container" podUID="ad0aee0e-6395-4067-a9da-9a5055c711cc" containerName="cleanup"
Nov 03 18:30:58.859889 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:58.859717    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad0aee0e-6395-4067-a9da-9a5055c711cc" containerName="cleanup"
Nov 03 18:30:58.859889 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:58.859727    5315 cpu_manager.go:414] "containerMap: removing container" podUID="ad0aee0e-6395-4067-a9da-9a5055c711cc" containerName="cleanup"
Nov 03 18:30:58.859889 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:58.859733    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad0aee0e-6395-4067-a9da-9a5055c711cc" containerName="cleanup"
Nov 03 18:30:58.859889 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:58.859796    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad0aee0e-6395-4067-a9da-9a5055c711cc" containerName="cleanup"
Nov 03 18:30:58.859889 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:58.859811    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad0aee0e-6395-4067-a9da-9a5055c711cc" containerName="cleanup"
Nov 03 18:30:58.862815 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:58.862785    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-lm6tb"
Nov 03 18:30:58.911355 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:58.911322    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:30:58.937659 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:58.937626    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5zcc\" (UniqueName: \"kubernetes.io/projected/9b92e48f-b943-4252-b487-7af6a4c8e8be-kube-api-access-w5zcc\") pod \"maas-api-revoke-sub-d576e68b-lm6tb\" (UID: \"9b92e48f-b943-4252-b487-7af6a4c8e8be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-lm6tb"
Nov 03 18:30:58.937659 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:58.937667    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b92e48f-b943-4252-b487-7af6a4c8e8be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-lm6tb\" (UID: \"9b92e48f-b943-4252-b487-7af6a4c8e8be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-lm6tb"
Nov 03 18:30:58.937944 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:58.937705    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b92e48f-b943-4252-b487-7af6a4c8e8be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-lm6tb\" (UID: \"9b92e48f-b943-4252-b487-7af6a4c8e8be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-lm6tb"
Nov 03 18:30:58.957390 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:58.957350    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:30:58.957390 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:58.957358    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:30:58.957620 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:58.957411    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:30:58.959845 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:58.959821    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-lm6tb"]
Nov 03 18:30:59.038216 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:59.038174    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b92e48f-b943-4252-b487-7af6a4c8e8be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-lm6tb\" (UID: \"9b92e48f-b943-4252-b487-7af6a4c8e8be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-lm6tb"
Nov 03 18:30:59.038400 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:59.038236    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b92e48f-b943-4252-b487-7af6a4c8e8be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-lm6tb\" (UID: \"9b92e48f-b943-4252-b487-7af6a4c8e8be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-lm6tb"
Nov 03 18:30:59.038400 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:59.038367    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5zcc\" (UniqueName: \"kubernetes.io/projected/9b92e48f-b943-4252-b487-7af6a4c8e8be-kube-api-access-w5zcc\") pod \"maas-api-revoke-sub-d576e68b-lm6tb\" (UID: \"9b92e48f-b943-4252-b487-7af6a4c8e8be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-lm6tb"
Nov 03 18:30:59.038948 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:59.038928    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b92e48f-b943-4252-b487-7af6a4c8e8be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-lm6tb\" (UID: \"9b92e48f-b943-4252-b487-7af6a4c8e8be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-lm6tb"
Nov 03 18:30:59.040623 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:59.040594    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b92e48f-b943-4252-b487-7af6a4c8e8be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-lm6tb\" (UID: \"9b92e48f-b943-4252-b487-7af6a4c8e8be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-lm6tb"
Nov 03 18:30:59.095464 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:59.095428    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5zcc\" (UniqueName: \"kubernetes.io/projected/9b92e48f-b943-4252-b487-7af6a4c8e8be-kube-api-access-w5zcc\") pod \"maas-api-revoke-sub-d576e68b-lm6tb\" (UID: \"9b92e48f-b943-4252-b487-7af6a4c8e8be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-lm6tb"
Nov 03 18:30:59.175634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:59.175527    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-lm6tb"
Nov 03 18:30:59.544789 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:59.544748    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-lm6tb"]
Nov 03 18:30:59.552123 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:30:59.552094    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b92e48f_b943_4252_b487_7af6a4c8e8be.slice/crio-c20945f6d7b605eb18b4f061e7684b2b3b01069758d234548346e7e10f483836 WatchSource:0}: Error finding container c20945f6d7b605eb18b4f061e7684b2b3b01069758d234548346e7e10f483836: Status 404 returned error can't find the container with id c20945f6d7b605eb18b4f061e7684b2b3b01069758d234548346e7e10f483836
Nov 03 18:30:59.886408 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:59.886373    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-lm6tb" event={"ID":"9b92e48f-b943-4252-b487-7af6a4c8e8be","Type":"ContainerStarted","Data":"349ff2e01710edb58b42adfe29eaa95ee041a78a6319a3f2f914caeaed1823cd"}
Nov 03 18:30:59.886911 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:59.886436    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-lm6tb" event={"ID":"9b92e48f-b943-4252-b487-7af6a4c8e8be","Type":"ContainerStarted","Data":"c20945f6d7b605eb18b4f061e7684b2b3b01069758d234548346e7e10f483836"}
Nov 03 18:30:59.995241 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:30:59.995180    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-lm6tb" podStartSLOduration=1.9951648130000001 podStartE2EDuration="1.995164813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:30:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:30:59.99388321 +0000 UTC m=+2699.010081279" watchObservedRunningTime="2026-11-03 18:30:59.995164813 +0000 UTC m=+2699.011362928"
Nov 03 18:31:00.894961 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:00.894922    5315 generic.go:356] "Generic (PLEG): container finished" podID="9b92e48f-b943-4252-b487-7af6a4c8e8be" containerID="349ff2e01710edb58b42adfe29eaa95ee041a78a6319a3f2f914caeaed1823cd" exitCode=0
Nov 03 18:31:00.895369 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:00.894960    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-lm6tb" event={"ID":"9b92e48f-b943-4252-b487-7af6a4c8e8be","Type":"ContainerDied","Data":"349ff2e01710edb58b42adfe29eaa95ee041a78a6319a3f2f914caeaed1823cd"}
Nov 03 18:31:01.689203 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:01.689174    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-97p65_c29d200a-c917-46e5-9cf5-21f4d952e470/ovn-acl-logging/0.log"
Nov 03 18:31:01.733506 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:01.733476    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-97p65_c29d200a-c917-46e5-9cf5-21f4d952e470/ovn-acl-logging/0.log"
Nov 03 18:31:02.036212 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:02.036183    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-lm6tb"
Nov 03 18:31:02.068708 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:02.068666    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b92e48f-b943-4252-b487-7af6a4c8e8be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b92e48f-b943-4252-b487-7af6a4c8e8be-maas-api-xks-ca\") pod \"9b92e48f-b943-4252-b487-7af6a4c8e8be\" (UID: \"9b92e48f-b943-4252-b487-7af6a4c8e8be\") "
Nov 03 18:31:02.068920 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:02.068754    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b92e48f-b943-4252-b487-7af6a4c8e8be-kube-api-access-w5zcc\" (UniqueName: \"kubernetes.io/projected/9b92e48f-b943-4252-b487-7af6a4c8e8be-kube-api-access-w5zcc\") pod \"9b92e48f-b943-4252-b487-7af6a4c8e8be\" (UID: \"9b92e48f-b943-4252-b487-7af6a4c8e8be\") "
Nov 03 18:31:02.068920 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:02.068779    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b92e48f-b943-4252-b487-7af6a4c8e8be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b92e48f-b943-4252-b487-7af6a4c8e8be-maas-api-openshift-ca\") pod \"9b92e48f-b943-4252-b487-7af6a4c8e8be\" (UID: \"9b92e48f-b943-4252-b487-7af6a4c8e8be\") "
Nov 03 18:31:02.069168 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:02.069144    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b92e48f-b943-4252-b487-7af6a4c8e8be-maas-api-openshift-ca" pod "9b92e48f-b943-4252-b487-7af6a4c8e8be" (UID: "9b92e48f-b943-4252-b487-7af6a4c8e8be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:31:02.070963 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:02.070935    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b92e48f-b943-4252-b487-7af6a4c8e8be-maas-api-xks-ca" pod "9b92e48f-b943-4252-b487-7af6a4c8e8be" (UID: "9b92e48f-b943-4252-b487-7af6a4c8e8be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:31:02.071065 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:02.070958    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b92e48f-b943-4252-b487-7af6a4c8e8be-kube-api-access-w5zcc" pod "9b92e48f-b943-4252-b487-7af6a4c8e8be" (UID: "9b92e48f-b943-4252-b487-7af6a4c8e8be"). InnerVolumeSpecName "kube-api-access-w5zcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:31:02.169996 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:02.169951    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b92e48f-b943-4252-b487-7af6a4c8e8be-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:31:02.169996 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:02.169993    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5zcc\" (UniqueName: \"kubernetes.io/projected/9b92e48f-b943-4252-b487-7af6a4c8e8be-kube-api-access-w5zcc\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:31:02.169996 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:02.170003    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b92e48f-b943-4252-b487-7af6a4c8e8be-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:31:02.905731 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:02.905698    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-lm6tb"
Nov 03 18:31:02.905934 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:02.905742    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-lm6tb" event={"ID":"9b92e48f-b943-4252-b487-7af6a4c8e8be","Type":"ContainerDied","Data":"c20945f6d7b605eb18b4f061e7684b2b3b01069758d234548346e7e10f483836"}
Nov 03 18:31:02.905934 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:02.905791    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c20945f6d7b605eb18b4f061e7684b2b3b01069758d234548346e7e10f483836"
Nov 03 18:31:06.082477 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:06.082429    5315 scope.go:122] "RemoveContainer" containerID="15b0b35ee20019a7a6c39fe92da43ab28892948be81c33508b7226e01fb51c32"
Nov 03 18:31:06.090960 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:06.090938    5315 scope.go:122] "RemoveContainer" containerID="f63fa19daf59a1965e284bf4f0fa5d99d47e397bf22a81a23bd1faed4f856492"
Nov 03 18:31:23.044070 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:23.043986    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-r557b"]
Nov 03 18:31:23.089890 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:23.089856    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-r557b"]
Nov 03 18:31:23.106205 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:23.106172    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ee1ca3d-1df0-408d-9c64-158a7e662e4d" path="/var/lib/kubelet/pods/0ee1ca3d-1df0-408d-9c64-158a7e662e4d/volumes"
Nov 03 18:31:24.577012 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:31:24.576972    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:31:28.046801 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:28.046766    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6ndfn"]
Nov 03 18:31:28.047207 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:28.047128    5315 cpu_manager.go:414] "containerMap: removing container" podUID="9b92e48f-b943-4252-b487-7af6a4c8e8be" containerName="revoke-keys"
Nov 03 18:31:28.047207 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:28.047140    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b92e48f-b943-4252-b487-7af6a4c8e8be" containerName="revoke-keys"
Nov 03 18:31:28.047207 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:28.047193    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad0aee0e-6395-4067-a9da-9a5055c711cc" containerName="cleanup"
Nov 03 18:31:28.047207 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:28.047204    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b92e48f-b943-4252-b487-7af6a4c8e8be" containerName="revoke-keys"
Nov 03 18:31:28.051746 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:28.051727    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6ndfn"
Nov 03 18:31:28.146243 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:28.146201    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:31:28.146464 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:28.146245    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:31:28.146464 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:28.146245    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:31:28.146464 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:28.146327    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:31:28.147645 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:28.147622    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6ndfn"]
Nov 03 18:31:28.156777 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:28.156752    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r64nr\" (UniqueName: \"kubernetes.io/projected/36719fa9-a3fc-4308-91b8-b41cd779725a-kube-api-access-r64nr\") pod \"maas-api-revoke-sub-d2bc93bc-6ndfn\" (UID: \"36719fa9-a3fc-4308-91b8-b41cd779725a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6ndfn"
Nov 03 18:31:28.156885 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:28.156822    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36719fa9-a3fc-4308-91b8-b41cd779725a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-6ndfn\" (UID: \"36719fa9-a3fc-4308-91b8-b41cd779725a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6ndfn"
Nov 03 18:31:28.156931 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:28.156889    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36719fa9-a3fc-4308-91b8-b41cd779725a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-6ndfn\" (UID: \"36719fa9-a3fc-4308-91b8-b41cd779725a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6ndfn"
Nov 03 18:31:28.257862 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:28.257824    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36719fa9-a3fc-4308-91b8-b41cd779725a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-6ndfn\" (UID: \"36719fa9-a3fc-4308-91b8-b41cd779725a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6ndfn"
Nov 03 18:31:28.257862 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:28.257866    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36719fa9-a3fc-4308-91b8-b41cd779725a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-6ndfn\" (UID: \"36719fa9-a3fc-4308-91b8-b41cd779725a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6ndfn"
Nov 03 18:31:28.258177 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:28.257908    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r64nr\" (UniqueName: \"kubernetes.io/projected/36719fa9-a3fc-4308-91b8-b41cd779725a-kube-api-access-r64nr\") pod \"maas-api-revoke-sub-d2bc93bc-6ndfn\" (UID: \"36719fa9-a3fc-4308-91b8-b41cd779725a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6ndfn"
Nov 03 18:31:28.258525 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:28.258495    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36719fa9-a3fc-4308-91b8-b41cd779725a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-6ndfn\" (UID: \"36719fa9-a3fc-4308-91b8-b41cd779725a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6ndfn"
Nov 03 18:31:28.260273 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:28.260243    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36719fa9-a3fc-4308-91b8-b41cd779725a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-6ndfn\" (UID: \"36719fa9-a3fc-4308-91b8-b41cd779725a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6ndfn"
Nov 03 18:31:28.314826 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:28.314745    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r64nr\" (UniqueName: \"kubernetes.io/projected/36719fa9-a3fc-4308-91b8-b41cd779725a-kube-api-access-r64nr\") pod \"maas-api-revoke-sub-d2bc93bc-6ndfn\" (UID: \"36719fa9-a3fc-4308-91b8-b41cd779725a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6ndfn"
Nov 03 18:31:28.364735 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:28.364690    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6ndfn"
Nov 03 18:31:28.729632 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:28.729534    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6ndfn"]
Nov 03 18:31:28.736627 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:31:28.736585    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36719fa9_a3fc_4308_91b8_b41cd779725a.slice/crio-7ac8b186314fc5f5620eafe20be5c583839030461c0bf583cffa55790e655e0c WatchSource:0}: Error finding container 7ac8b186314fc5f5620eafe20be5c583839030461c0bf583cffa55790e655e0c: Status 404 returned error can't find the container with id 7ac8b186314fc5f5620eafe20be5c583839030461c0bf583cffa55790e655e0c
Nov 03 18:31:29.050336 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:29.050305    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6ndfn" event={"ID":"36719fa9-a3fc-4308-91b8-b41cd779725a","Type":"ContainerStarted","Data":"4cb9accc3c1b0238de3cb32a9c6be820aacc07f5e27eecbcbb1fb2c4c3c86086"}
Nov 03 18:31:29.050694 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:29.050355    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6ndfn" event={"ID":"36719fa9-a3fc-4308-91b8-b41cd779725a","Type":"ContainerStarted","Data":"7ac8b186314fc5f5620eafe20be5c583839030461c0bf583cffa55790e655e0c"}
Nov 03 18:31:29.158063 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:29.158006    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6ndfn" podStartSLOduration=1.157992226 podStartE2EDuration="1.157992226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:31:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:31:29.157416524 +0000 UTC m=+2728.173614593" watchObservedRunningTime="2026-11-03 18:31:29.157992226 +0000 UTC m=+2728.174190289"
Nov 03 18:31:30.056595 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:30.056563    5315 generic.go:356] "Generic (PLEG): container finished" podID="36719fa9-a3fc-4308-91b8-b41cd779725a" containerID="4cb9accc3c1b0238de3cb32a9c6be820aacc07f5e27eecbcbb1fb2c4c3c86086" exitCode=0
Nov 03 18:31:30.056595 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:30.056600    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6ndfn" event={"ID":"36719fa9-a3fc-4308-91b8-b41cd779725a","Type":"ContainerDied","Data":"4cb9accc3c1b0238de3cb32a9c6be820aacc07f5e27eecbcbb1fb2c4c3c86086"}
Nov 03 18:31:31.206425 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:31.206242    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6ndfn"
Nov 03 18:31:31.388394 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:31.388294    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36719fa9-a3fc-4308-91b8-b41cd779725a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36719fa9-a3fc-4308-91b8-b41cd779725a-maas-api-openshift-ca\") pod \"36719fa9-a3fc-4308-91b8-b41cd779725a\" (UID: \"36719fa9-a3fc-4308-91b8-b41cd779725a\") "
Nov 03 18:31:31.388582 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:31.388402    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36719fa9-a3fc-4308-91b8-b41cd779725a-kube-api-access-r64nr\" (UniqueName: \"kubernetes.io/projected/36719fa9-a3fc-4308-91b8-b41cd779725a-kube-api-access-r64nr\") pod \"36719fa9-a3fc-4308-91b8-b41cd779725a\" (UID: \"36719fa9-a3fc-4308-91b8-b41cd779725a\") "
Nov 03 18:31:31.388582 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:31.388439    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36719fa9-a3fc-4308-91b8-b41cd779725a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36719fa9-a3fc-4308-91b8-b41cd779725a-maas-api-xks-ca\") pod \"36719fa9-a3fc-4308-91b8-b41cd779725a\" (UID: \"36719fa9-a3fc-4308-91b8-b41cd779725a\") "
Nov 03 18:31:31.388782 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:31.388752    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36719fa9-a3fc-4308-91b8-b41cd779725a-maas-api-openshift-ca" pod "36719fa9-a3fc-4308-91b8-b41cd779725a" (UID: "36719fa9-a3fc-4308-91b8-b41cd779725a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:31:31.390526 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:31.390502    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36719fa9-a3fc-4308-91b8-b41cd779725a-maas-api-xks-ca" pod "36719fa9-a3fc-4308-91b8-b41cd779725a" (UID: "36719fa9-a3fc-4308-91b8-b41cd779725a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:31:31.390631 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:31.390505    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36719fa9-a3fc-4308-91b8-b41cd779725a-kube-api-access-r64nr" pod "36719fa9-a3fc-4308-91b8-b41cd779725a" (UID: "36719fa9-a3fc-4308-91b8-b41cd779725a"). InnerVolumeSpecName "kube-api-access-r64nr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:31:31.490225 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:31.490165    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r64nr\" (UniqueName: \"kubernetes.io/projected/36719fa9-a3fc-4308-91b8-b41cd779725a-kube-api-access-r64nr\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:31:31.490225 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:31.490220    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36719fa9-a3fc-4308-91b8-b41cd779725a-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:31:31.490225 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:31.490235    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36719fa9-a3fc-4308-91b8-b41cd779725a-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:31:32.068404 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:32.068373    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6ndfn"
Nov 03 18:31:32.068584 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:32.068407    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6ndfn" event={"ID":"36719fa9-a3fc-4308-91b8-b41cd779725a","Type":"ContainerDied","Data":"7ac8b186314fc5f5620eafe20be5c583839030461c0bf583cffa55790e655e0c"}
Nov 03 18:31:32.068584 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:32.068478    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ac8b186314fc5f5620eafe20be5c583839030461c0bf583cffa55790e655e0c"
Nov 03 18:31:33.045092 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:33.045056    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-gttwx"]
Nov 03 18:31:33.091119 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:33.091089    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-gttwx"]
Nov 03 18:31:33.102069 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:33.102036    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3" path="/var/lib/kubelet/pods/ca5a2d37-4b8b-4b10-b5a0-d4edc802f0a3/volumes"
Nov 03 18:31:41.140782 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:41.140740    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62"]
Nov 03 18:31:41.141292 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:41.141271    5315 cpu_manager.go:414] "containerMap: removing container" podUID="36719fa9-a3fc-4308-91b8-b41cd779725a" containerName="revoke-keys"
Nov 03 18:31:41.141372 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:41.141291    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36719fa9-a3fc-4308-91b8-b41cd779725a" containerName="revoke-keys"
Nov 03 18:31:41.141429 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:41.141384    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="36719fa9-a3fc-4308-91b8-b41cd779725a" containerName="revoke-keys"
Nov 03 18:31:41.145252 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:41.145221    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62"
Nov 03 18:31:41.174193 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:41.174156    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49e30524-b5c2-4986-a24a-40d31a741085-istio-podinfo\") pod \"e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62\" (UID: \"49e30524-b5c2-4986-a24a-40d31a741085\") " pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62"
Nov 03 18:31:41.174193 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:41.174191    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49e30524-b5c2-4986-a24a-40d31a741085-istiod-ca-cert\") pod \"e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62\" (UID: \"49e30524-b5c2-4986-a24a-40d31a741085\") " pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62"
Nov 03 18:31:41.174412 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:41.174213    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e30524-b5c2-4986-a24a-40d31a741085-credential-socket\") pod \"e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62\" (UID: \"49e30524-b5c2-4986-a24a-40d31a741085\") " pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62"
Nov 03 18:31:41.174412 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:41.174327    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szkwg\" (UniqueName: \"kubernetes.io/projected/49e30524-b5c2-4986-a24a-40d31a741085-kube-api-access-szkwg\") pod \"e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62\" (UID: \"49e30524-b5c2-4986-a24a-40d31a741085\") " pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62"
Nov 03 18:31:41.174412 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:41.174366    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49e30524-b5c2-4986-a24a-40d31a741085-istio-envoy\") pod \"e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62\" (UID: \"49e30524-b5c2-4986-a24a-40d31a741085\") " pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62"
Nov 03 18:31:41.174412 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:41.174405    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e30524-b5c2-4986-a24a-40d31a741085-workload-socket\") pod \"e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62\" (UID: \"49e30524-b5c2-4986-a24a-40d31a741085\") " pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62"
Nov 03 18:31:41.174639 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:41.174597    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49e30524-b5c2-4986-a24a-40d31a741085-istio-data\") pod \"e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62\" (UID: \"49e30524-b5c2-4986-a24a-40d31a741085\") " pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62"
Nov 03 18:31:41.174715 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:41.174687    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49e30524-b5c2-4986-a24a-40d31a741085-workload-certs\") pod \"e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62\" (UID: \"49e30524-b5c2-4986-a24a-40d31a741085\") " pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62"
Nov 03 18:31:41.174787 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:41.174723    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49e30524-b5c2-4986-a24a-40d31a741085-istio-token\") pod \"e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62\" (UID: \"49e30524-b5c2-4986-a24a-40d31a741085\") " pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62"
Nov 03 18:31:41.222022 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:41.221974    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-0d7aec-openshift-default-dockercfg-6p5d7\""
Nov 03 18:31:41.261542 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:41.261506    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62"]
Nov 03 18:31:41.276353 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:41.276313    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e30524-b5c2-4986-a24a-40d31a741085-workload-socket\") pod \"e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62\" (UID: \"49e30524-b5c2-4986-a24a-40d31a741085\") " pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62"
Nov 03 18:31:41.276630 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:41.276589    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49e30524-b5c2-4986-a24a-40d31a741085-istio-data\") pod \"e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62\" (UID: \"49e30524-b5c2-4986-a24a-40d31a741085\") " pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62"
Nov 03 18:31:41.276772 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:41.276669    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49e30524-b5c2-4986-a24a-40d31a741085-workload-certs\") pod \"e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62\" (UID: \"49e30524-b5c2-4986-a24a-40d31a741085\") " pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62"
Nov 03 18:31:41.276772 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:41.276697    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49e30524-b5c2-4986-a24a-40d31a741085-istio-token\") pod \"e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62\" (UID: \"49e30524-b5c2-4986-a24a-40d31a741085\") " pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62"
Nov 03 18:31:41.276772 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:41.276732    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49e30524-b5c2-4986-a24a-40d31a741085-istio-podinfo\") pod \"e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62\" (UID: \"49e30524-b5c2-4986-a24a-40d31a741085\") " pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62"
Nov 03 18:31:41.276772 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:41.276758    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49e30524-b5c2-4986-a24a-40d31a741085-istiod-ca-cert\") pod \"e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62\" (UID: \"49e30524-b5c2-4986-a24a-40d31a741085\") " pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62"
Nov 03 18:31:41.277026 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:41.276793    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e30524-b5c2-4986-a24a-40d31a741085-credential-socket\") pod \"e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62\" (UID: \"49e30524-b5c2-4986-a24a-40d31a741085\") " pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62"
Nov 03 18:31:41.277026 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:41.276845    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szkwg\" (UniqueName: \"kubernetes.io/projected/49e30524-b5c2-4986-a24a-40d31a741085-kube-api-access-szkwg\") pod \"e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62\" (UID: \"49e30524-b5c2-4986-a24a-40d31a741085\") " pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62"
Nov 03 18:31:41.277026 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:41.276908    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49e30524-b5c2-4986-a24a-40d31a741085-istio-envoy\") pod \"e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62\" (UID: \"49e30524-b5c2-4986-a24a-40d31a741085\") " pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62"
Nov 03 18:31:41.277996 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:41.277958    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49e30524-b5c2-4986-a24a-40d31a741085-workload-certs\") pod \"e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62\" (UID: \"49e30524-b5c2-4986-a24a-40d31a741085\") " pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62"
Nov 03 18:31:41.278224 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:41.278206    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e30524-b5c2-4986-a24a-40d31a741085-workload-socket\") pod \"e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62\" (UID: \"49e30524-b5c2-4986-a24a-40d31a741085\") " pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62"
Nov 03 18:31:41.278863 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:41.278837    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49e30524-b5c2-4986-a24a-40d31a741085-istiod-ca-cert\") pod \"e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62\" (UID: \"49e30524-b5c2-4986-a24a-40d31a741085\") " pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62"
Nov 03 18:31:41.279806 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:41.279295    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e30524-b5c2-4986-a24a-40d31a741085-credential-socket\") pod \"e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62\" (UID: \"49e30524-b5c2-4986-a24a-40d31a741085\") " pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62"
Nov 03 18:31:41.280253 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:41.280232    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49e30524-b5c2-4986-a24a-40d31a741085-istio-data\") pod \"e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62\" (UID: \"49e30524-b5c2-4986-a24a-40d31a741085\") " pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62"
Nov 03 18:31:41.282862 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:41.282837    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49e30524-b5c2-4986-a24a-40d31a741085-istio-podinfo\") pod \"e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62\" (UID: \"49e30524-b5c2-4986-a24a-40d31a741085\") " pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62"
Nov 03 18:31:41.288138 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:41.288109    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49e30524-b5c2-4986-a24a-40d31a741085-istio-envoy\") pod \"e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62\" (UID: \"49e30524-b5c2-4986-a24a-40d31a741085\") " pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62"
Nov 03 18:31:41.355754 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:41.355726    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szkwg\" (UniqueName: \"kubernetes.io/projected/49e30524-b5c2-4986-a24a-40d31a741085-kube-api-access-szkwg\") pod \"e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62\" (UID: \"49e30524-b5c2-4986-a24a-40d31a741085\") " pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62"
Nov 03 18:31:41.356308 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:41.356291    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49e30524-b5c2-4986-a24a-40d31a741085-istio-token\") pod \"e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62\" (UID: \"49e30524-b5c2-4986-a24a-40d31a741085\") " pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62"
Nov 03 18:31:41.457707 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:41.457601    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62"
Nov 03 18:31:41.889944 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:41.889912    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62"]
Nov 03 18:31:41.897557 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:31:41.897521    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49e30524_b5c2_4986_a24a_40d31a741085.slice/crio-d79f237aa1cf6017e75c22f00c9281ff5786f62bc9a28dc2082a760454729db5 WatchSource:0}: Error finding container d79f237aa1cf6017e75c22f00c9281ff5786f62bc9a28dc2082a760454729db5: Status 404 returned error can't find the container with id d79f237aa1cf6017e75c22f00c9281ff5786f62bc9a28dc2082a760454729db5
Nov 03 18:31:41.899774 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:41.899741    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:31:41.899856 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:41.899812    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:31:41.899856 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:41.899846    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:31:42.116277 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:42.116227    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62" event={"ID":"49e30524-b5c2-4986-a24a-40d31a741085","Type":"ContainerStarted","Data":"7e060bd643a568e827e7b8510392cc1467e46d84c41d4e00ef27994f8725371e"}
Nov 03 18:31:42.116474 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:42.116307    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62" event={"ID":"49e30524-b5c2-4986-a24a-40d31a741085","Type":"ContainerStarted","Data":"d79f237aa1cf6017e75c22f00c9281ff5786f62bc9a28dc2082a760454729db5"}
Nov 03 18:31:42.230792 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:42.230684    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62" podStartSLOduration=1.2306687410000001 podStartE2EDuration="1.230668741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:31:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:31:42.228839283 +0000 UTC m=+2741.245037352" watchObservedRunningTime="2026-11-03 18:31:42.230668741 +0000 UTC m=+2741.246866809"
Nov 03 18:31:42.458789 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:42.458748    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62"
Nov 03 18:31:42.463725 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:42.463699    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62"
Nov 03 18:31:43.121089 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:43.121055    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62"
Nov 03 18:31:43.122135 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:31:43.122112    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62"
Nov 03 18:31:54.579210 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:31:54.579171    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:32:06.117263 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:06.117229    5315 scope.go:122] "RemoveContainer" containerID="7643a7e8fa25648c0f70a7440b61b390220e19b4c2d73252cad3dd77965d754a"
Nov 03 18:32:06.151301 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:06.151273    5315 scope.go:122] "RemoveContainer" containerID="1ddbc662389ac4fc92fe01a6a96c9db09e50e8a83d29142f9614c38b7956e452"
Nov 03 18:32:06.159667 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:06.159642    5315 scope.go:122] "RemoveContainer" containerID="e82eeb9313d7715ea9b1af20cf2694dc31d94306ec49637b7542bbd6a7b0ad23"
Nov 03 18:32:10.274296 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:10.274254    5315 generic.go:356] "Generic (PLEG): container finished" podID="49e30524-b5c2-4986-a24a-40d31a741085" containerID="7e060bd643a568e827e7b8510392cc1467e46d84c41d4e00ef27994f8725371e" exitCode=0
Nov 03 18:32:10.274791 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:10.274331    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62" event={"ID":"49e30524-b5c2-4986-a24a-40d31a741085","Type":"ContainerDied","Data":"7e060bd643a568e827e7b8510392cc1467e46d84c41d4e00ef27994f8725371e"}
Nov 03 18:32:10.274791 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:10.274762    5315 scope.go:122] "RemoveContainer" containerID="7e060bd643a568e827e7b8510392cc1467e46d84c41d4e00ef27994f8725371e"
Nov 03 18:32:10.275940 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:10.275911    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:32:10.276001 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:10.275981    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:32:10.276038 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:10.276014    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:32:11.281196 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:11.281157    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62" event={"ID":"49e30524-b5c2-4986-a24a-40d31a741085","Type":"ContainerStarted","Data":"b5f29d313ff5abb089d1376e5ce522dde1fd2c9356baa74efef493f164fe4e93"}
Nov 03 18:32:11.458043 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:11.458007    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62"
Nov 03 18:32:11.459307 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:11.459278    5315 patch_prober.go:29] interesting pod/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body=
Nov 03 18:32:11.459431 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:11.459337    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62" podUID="49e30524-b5c2-4986-a24a-40d31a741085" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused"
Nov 03 18:32:12.458547 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:12.458504    5315 patch_prober.go:29] interesting pod/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body=
Nov 03 18:32:12.458973 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:12.458595    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62" podUID="49e30524-b5c2-4986-a24a-40d31a741085" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused"
Nov 03 18:32:13.121540 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:13.121509    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62"
Nov 03 18:32:13.458082 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:13.457983    5315 patch_prober.go:29] interesting pod/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body=
Nov 03 18:32:13.458082 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:13.458041    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62" podUID="49e30524-b5c2-4986-a24a-40d31a741085" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused"
Nov 03 18:32:13.833310 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:13.833268    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-0d7aec/simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z"]
Nov 03 18:32:13.850652 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:13.849797    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-0d7aec/simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z"
Nov 03 18:32:13.897326 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:13.897286    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-0d7aec/simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z"]
Nov 03 18:32:13.949048 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:13.948956    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-0d7aec\"/\"kube-root-ca.crt\""
Nov 03 18:32:13.949604 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:13.949572    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-0d7aec\"/\"openshift-service-ca.crt\""
Nov 03 18:32:13.949896 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:13.949876    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-0d7aec\"/\"simulator-single-main-0d7aec-kserve-self-signed-certs\""
Nov 03 18:32:13.950367 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:13.950338    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-0d7aec\"/\"default-dockercfg-bxdf4\""
Nov 03 18:32:13.958920 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:13.958885    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1eed39a-f722-4b9e-8c36-c591b39c58d5-home\") pod \"simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z\" (UID: \"a1eed39a-f722-4b9e-8c36-c591b39c58d5\") " pod="e2e-models-e2e-worker-single-main-0d7aec/simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z"
Nov 03 18:32:13.959077 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:13.958953    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1eed39a-f722-4b9e-8c36-c591b39c58d5-model-cache\") pod \"simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z\" (UID: \"a1eed39a-f722-4b9e-8c36-c591b39c58d5\") " pod="e2e-models-e2e-worker-single-main-0d7aec/simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z"
Nov 03 18:32:13.959077 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:13.958996    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1eed39a-f722-4b9e-8c36-c591b39c58d5-tmp-dir\") pod \"simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z\" (UID: \"a1eed39a-f722-4b9e-8c36-c591b39c58d5\") " pod="e2e-models-e2e-worker-single-main-0d7aec/simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z"
Nov 03 18:32:13.959190 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:13.959163    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1eed39a-f722-4b9e-8c36-c591b39c58d5-dshm\") pod \"simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z\" (UID: \"a1eed39a-f722-4b9e-8c36-c591b39c58d5\") " pod="e2e-models-e2e-worker-single-main-0d7aec/simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z"
Nov 03 18:32:13.959315 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:13.959294    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1eed39a-f722-4b9e-8c36-c591b39c58d5-tls-certs\") pod \"simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z\" (UID: \"a1eed39a-f722-4b9e-8c36-c591b39c58d5\") " pod="e2e-models-e2e-worker-single-main-0d7aec/simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z"
Nov 03 18:32:13.959365 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:13.959328    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2q4s\" (UniqueName: \"kubernetes.io/projected/a1eed39a-f722-4b9e-8c36-c591b39c58d5-kube-api-access-v2q4s\") pod \"simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z\" (UID: \"a1eed39a-f722-4b9e-8c36-c591b39c58d5\") " pod="e2e-models-e2e-worker-single-main-0d7aec/simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z"
Nov 03 18:32:14.060198 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:14.060159    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1eed39a-f722-4b9e-8c36-c591b39c58d5-home\") pod \"simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z\" (UID: \"a1eed39a-f722-4b9e-8c36-c591b39c58d5\") " pod="e2e-models-e2e-worker-single-main-0d7aec/simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z"
Nov 03 18:32:14.060198 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:14.060200    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1eed39a-f722-4b9e-8c36-c591b39c58d5-model-cache\") pod \"simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z\" (UID: \"a1eed39a-f722-4b9e-8c36-c591b39c58d5\") " pod="e2e-models-e2e-worker-single-main-0d7aec/simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z"
Nov 03 18:32:14.060444 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:14.060227    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1eed39a-f722-4b9e-8c36-c591b39c58d5-tmp-dir\") pod \"simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z\" (UID: \"a1eed39a-f722-4b9e-8c36-c591b39c58d5\") " pod="e2e-models-e2e-worker-single-main-0d7aec/simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z"
Nov 03 18:32:14.060444 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:14.060275    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1eed39a-f722-4b9e-8c36-c591b39c58d5-dshm\") pod \"simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z\" (UID: \"a1eed39a-f722-4b9e-8c36-c591b39c58d5\") " pod="e2e-models-e2e-worker-single-main-0d7aec/simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z"
Nov 03 18:32:14.060444 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:14.060331    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1eed39a-f722-4b9e-8c36-c591b39c58d5-tls-certs\") pod \"simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z\" (UID: \"a1eed39a-f722-4b9e-8c36-c591b39c58d5\") " pod="e2e-models-e2e-worker-single-main-0d7aec/simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z"
Nov 03 18:32:14.060444 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:14.060355    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2q4s\" (UniqueName: \"kubernetes.io/projected/a1eed39a-f722-4b9e-8c36-c591b39c58d5-kube-api-access-v2q4s\") pod \"simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z\" (UID: \"a1eed39a-f722-4b9e-8c36-c591b39c58d5\") " pod="e2e-models-e2e-worker-single-main-0d7aec/simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z"
Nov 03 18:32:14.060715 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:14.060570    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1eed39a-f722-4b9e-8c36-c591b39c58d5-home\") pod \"simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z\" (UID: \"a1eed39a-f722-4b9e-8c36-c591b39c58d5\") " pod="e2e-models-e2e-worker-single-main-0d7aec/simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z"
Nov 03 18:32:14.060715 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:14.060655    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1eed39a-f722-4b9e-8c36-c591b39c58d5-model-cache\") pod \"simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z\" (UID: \"a1eed39a-f722-4b9e-8c36-c591b39c58d5\") " pod="e2e-models-e2e-worker-single-main-0d7aec/simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z"
Nov 03 18:32:14.060836 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:14.060731    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1eed39a-f722-4b9e-8c36-c591b39c58d5-tmp-dir\") pod \"simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z\" (UID: \"a1eed39a-f722-4b9e-8c36-c591b39c58d5\") " pod="e2e-models-e2e-worker-single-main-0d7aec/simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z"
Nov 03 18:32:14.062738 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:14.062714    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1eed39a-f722-4b9e-8c36-c591b39c58d5-dshm\") pod \"simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z\" (UID: \"a1eed39a-f722-4b9e-8c36-c591b39c58d5\") " pod="e2e-models-e2e-worker-single-main-0d7aec/simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z"
Nov 03 18:32:14.062900 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:14.062882    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1eed39a-f722-4b9e-8c36-c591b39c58d5-tls-certs\") pod \"simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z\" (UID: \"a1eed39a-f722-4b9e-8c36-c591b39c58d5\") " pod="e2e-models-e2e-worker-single-main-0d7aec/simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z"
Nov 03 18:32:14.118279 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:14.118203    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2q4s\" (UniqueName: \"kubernetes.io/projected/a1eed39a-f722-4b9e-8c36-c591b39c58d5-kube-api-access-v2q4s\") pod \"simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z\" (UID: \"a1eed39a-f722-4b9e-8c36-c591b39c58d5\") " pod="e2e-models-e2e-worker-single-main-0d7aec/simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z"
Nov 03 18:32:14.167122 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:14.167082    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-0d7aec/simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z"
Nov 03 18:32:14.458630 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:14.458507    5315 patch_prober.go:29] interesting pod/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body=
Nov 03 18:32:14.458630 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:14.458589    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62" podUID="49e30524-b5c2-4986-a24a-40d31a741085" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused"
Nov 03 18:32:14.550738 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:14.550685    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-0d7aec/simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z"]
Nov 03 18:32:14.552199 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:32:14.552167    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1eed39a_f722_4b9e_8c36_c591b39c58d5.slice/crio-afa5da9179fe5e992f6aec9915e418eec8dcfab8a20f45fe97eed6ad1bc2faad WatchSource:0}: Error finding container afa5da9179fe5e992f6aec9915e418eec8dcfab8a20f45fe97eed6ad1bc2faad: Status 404 returned error can't find the container with id afa5da9179fe5e992f6aec9915e418eec8dcfab8a20f45fe97eed6ad1bc2faad
Nov 03 18:32:15.307552 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:15.307497    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-0d7aec/simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z" event={"ID":"a1eed39a-f722-4b9e-8c36-c591b39c58d5","Type":"ContainerStarted","Data":"b4dc4939680411fe305fef906bf0c7badcee0056fcdfb89b42b5683ad40463a7"}
Nov 03 18:32:15.307552 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:15.307560    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-0d7aec/simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z" event={"ID":"a1eed39a-f722-4b9e-8c36-c591b39c58d5","Type":"ContainerStarted","Data":"afa5da9179fe5e992f6aec9915e418eec8dcfab8a20f45fe97eed6ad1bc2faad"}
Nov 03 18:32:15.417925 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:15.417870    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-0d7aec/simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z" podStartSLOduration=2.417855956 podStartE2EDuration="2.417855956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:32:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:32:15.416871258 +0000 UTC m=+2774.433069328" watchObservedRunningTime="2026-11-03 18:32:15.417855956 +0000 UTC m=+2774.434054024"
Nov 03 18:32:15.458228 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:15.458196    5315 patch_prober.go:29] interesting pod/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body=
Nov 03 18:32:15.458406 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:15.458253    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62" podUID="49e30524-b5c2-4986-a24a-40d31a741085" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused"
Nov 03 18:32:16.167895 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:16.167853    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-0d7aec/simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z"
Nov 03 18:32:16.462158 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:16.462082    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62"
Nov 03 18:32:16.463043 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:16.463019    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62"
Nov 03 18:32:24.168130 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:24.168089    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-0d7aec/simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z"
Nov 03 18:32:24.180724 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:24.180697    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-0d7aec/simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z"
Nov 03 18:32:24.369352 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:24.369321    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-0d7aec/simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z"
Nov 03 18:32:24.577522 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:32:24.577485    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:32:54.578331 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:32:54.578292    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:32:59.344797 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:59.344761    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73f4ae16-nkwrx"]
Nov 03 18:32:59.348343 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:59.348317    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73f4ae16-nkwrx"
Nov 03 18:32:59.396905 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:59.396862    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:32:59.413577 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:59.413541    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b52a2b0-d10c-4d06-9f78-e318f6a02973-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73f4ae16-nkwrx\" (UID: \"9b52a2b0-d10c-4d06-9f78-e318f6a02973\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73f4ae16-nkwrx"
Nov 03 18:32:59.413757 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:59.413597    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b52a2b0-d10c-4d06-9f78-e318f6a02973-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73f4ae16-nkwrx\" (UID: \"9b52a2b0-d10c-4d06-9f78-e318f6a02973\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73f4ae16-nkwrx"
Nov 03 18:32:59.413757 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:59.413740    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmsx5\" (UniqueName: \"kubernetes.io/projected/9b52a2b0-d10c-4d06-9f78-e318f6a02973-kube-api-access-cmsx5\") pod \"maas-api-revoke-sub-73f4ae16-nkwrx\" (UID: \"9b52a2b0-d10c-4d06-9f78-e318f6a02973\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73f4ae16-nkwrx"
Nov 03 18:32:59.443016 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:59.442982    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:32:59.443016 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:59.442981    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:32:59.443281 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:59.443238    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:32:59.444742 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:59.444718    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73f4ae16-nkwrx"]
Nov 03 18:32:59.515224 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:59.515188    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b52a2b0-d10c-4d06-9f78-e318f6a02973-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73f4ae16-nkwrx\" (UID: \"9b52a2b0-d10c-4d06-9f78-e318f6a02973\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73f4ae16-nkwrx"
Nov 03 18:32:59.515391 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:59.515244    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b52a2b0-d10c-4d06-9f78-e318f6a02973-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73f4ae16-nkwrx\" (UID: \"9b52a2b0-d10c-4d06-9f78-e318f6a02973\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73f4ae16-nkwrx"
Nov 03 18:32:59.515391 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:59.515315    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmsx5\" (UniqueName: \"kubernetes.io/projected/9b52a2b0-d10c-4d06-9f78-e318f6a02973-kube-api-access-cmsx5\") pod \"maas-api-revoke-sub-73f4ae16-nkwrx\" (UID: \"9b52a2b0-d10c-4d06-9f78-e318f6a02973\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73f4ae16-nkwrx"
Nov 03 18:32:59.515977 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:59.515955    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b52a2b0-d10c-4d06-9f78-e318f6a02973-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73f4ae16-nkwrx\" (UID: \"9b52a2b0-d10c-4d06-9f78-e318f6a02973\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73f4ae16-nkwrx"
Nov 03 18:32:59.517668 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:59.517646    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b52a2b0-d10c-4d06-9f78-e318f6a02973-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73f4ae16-nkwrx\" (UID: \"9b52a2b0-d10c-4d06-9f78-e318f6a02973\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73f4ae16-nkwrx"
Nov 03 18:32:59.572468 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:59.572433    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmsx5\" (UniqueName: \"kubernetes.io/projected/9b52a2b0-d10c-4d06-9f78-e318f6a02973-kube-api-access-cmsx5\") pod \"maas-api-revoke-sub-73f4ae16-nkwrx\" (UID: \"9b52a2b0-d10c-4d06-9f78-e318f6a02973\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73f4ae16-nkwrx"
Nov 03 18:32:59.670074 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:32:59.669997    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73f4ae16-nkwrx"
Nov 03 18:33:00.037664 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:00.037620    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73f4ae16-nkwrx"]
Nov 03 18:33:00.044323 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:33:00.044293    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b52a2b0_d10c_4d06_9f78_e318f6a02973.slice/crio-5bf07d8bac558dc524a58c10aa0aa7ce3eb8f026367ce25ff30a41e920599c96 WatchSource:0}: Error finding container 5bf07d8bac558dc524a58c10aa0aa7ce3eb8f026367ce25ff30a41e920599c96: Status 404 returned error can't find the container with id 5bf07d8bac558dc524a58c10aa0aa7ce3eb8f026367ce25ff30a41e920599c96
Nov 03 18:33:00.536320 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:00.536286    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73f4ae16-nkwrx" event={"ID":"9b52a2b0-d10c-4d06-9f78-e318f6a02973","Type":"ContainerStarted","Data":"e96775147cfc8e6df0ad4308917e212cbe42f9b68de2cbaa209cffe20e2467b7"}
Nov 03 18:33:00.536738 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:00.536344    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73f4ae16-nkwrx" event={"ID":"9b52a2b0-d10c-4d06-9f78-e318f6a02973","Type":"ContainerStarted","Data":"5bf07d8bac558dc524a58c10aa0aa7ce3eb8f026367ce25ff30a41e920599c96"}
Nov 03 18:33:00.645882 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:00.645782    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73f4ae16-nkwrx" podStartSLOduration=1.645767835 podStartE2EDuration="1.645767835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:32:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:33:00.644554016 +0000 UTC m=+2819.660752086" watchObservedRunningTime="2026-11-03 18:33:00.645767835 +0000 UTC m=+2819.661965903"
Nov 03 18:33:01.542878 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:01.542840    5315 generic.go:356] "Generic (PLEG): container finished" podID="9b52a2b0-d10c-4d06-9f78-e318f6a02973" containerID="e96775147cfc8e6df0ad4308917e212cbe42f9b68de2cbaa209cffe20e2467b7" exitCode=0
Nov 03 18:33:01.542878 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:01.542882    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73f4ae16-nkwrx" event={"ID":"9b52a2b0-d10c-4d06-9f78-e318f6a02973","Type":"ContainerDied","Data":"e96775147cfc8e6df0ad4308917e212cbe42f9b68de2cbaa209cffe20e2467b7"}
Nov 03 18:33:02.675745 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:02.675716    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73f4ae16-nkwrx"
Nov 03 18:33:02.759092 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:02.759058    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b52a2b0-d10c-4d06-9f78-e318f6a02973-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b52a2b0-d10c-4d06-9f78-e318f6a02973-maas-api-xks-ca\") pod \"9b52a2b0-d10c-4d06-9f78-e318f6a02973\" (UID: \"9b52a2b0-d10c-4d06-9f78-e318f6a02973\") "
Nov 03 18:33:02.759286 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:02.759165    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b52a2b0-d10c-4d06-9f78-e318f6a02973-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b52a2b0-d10c-4d06-9f78-e318f6a02973-maas-api-openshift-ca\") pod \"9b52a2b0-d10c-4d06-9f78-e318f6a02973\" (UID: \"9b52a2b0-d10c-4d06-9f78-e318f6a02973\") "
Nov 03 18:33:02.759286 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:02.759251    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b52a2b0-d10c-4d06-9f78-e318f6a02973-kube-api-access-cmsx5\" (UniqueName: \"kubernetes.io/projected/9b52a2b0-d10c-4d06-9f78-e318f6a02973-kube-api-access-cmsx5\") pod \"9b52a2b0-d10c-4d06-9f78-e318f6a02973\" (UID: \"9b52a2b0-d10c-4d06-9f78-e318f6a02973\") "
Nov 03 18:33:02.759534 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:02.759510    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b52a2b0-d10c-4d06-9f78-e318f6a02973-maas-api-openshift-ca" pod "9b52a2b0-d10c-4d06-9f78-e318f6a02973" (UID: "9b52a2b0-d10c-4d06-9f78-e318f6a02973"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:33:02.759685 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:02.759669    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b52a2b0-d10c-4d06-9f78-e318f6a02973-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:33:02.761265 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:02.761232    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b52a2b0-d10c-4d06-9f78-e318f6a02973-maas-api-xks-ca" pod "9b52a2b0-d10c-4d06-9f78-e318f6a02973" (UID: "9b52a2b0-d10c-4d06-9f78-e318f6a02973"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:33:02.761438 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:02.761416    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b52a2b0-d10c-4d06-9f78-e318f6a02973-kube-api-access-cmsx5" pod "9b52a2b0-d10c-4d06-9f78-e318f6a02973" (UID: "9b52a2b0-d10c-4d06-9f78-e318f6a02973"). InnerVolumeSpecName "kube-api-access-cmsx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:33:02.860652 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:02.860547    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmsx5\" (UniqueName: \"kubernetes.io/projected/9b52a2b0-d10c-4d06-9f78-e318f6a02973-kube-api-access-cmsx5\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:33:02.860652 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:02.860578    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b52a2b0-d10c-4d06-9f78-e318f6a02973-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:33:03.554291 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:03.554257    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73f4ae16-nkwrx"
Nov 03 18:33:03.554476 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:03.554260    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73f4ae16-nkwrx" event={"ID":"9b52a2b0-d10c-4d06-9f78-e318f6a02973","Type":"ContainerDied","Data":"5bf07d8bac558dc524a58c10aa0aa7ce3eb8f026367ce25ff30a41e920599c96"}
Nov 03 18:33:03.554476 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:03.554386    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5bf07d8bac558dc524a58c10aa0aa7ce3eb8f026367ce25ff30a41e920599c96"
Nov 03 18:33:06.185175 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:06.185146    5315 scope.go:122] "RemoveContainer" containerID="9623f8c329c41769ebcab81a008480baa381fa2669126d9da47b8d157d6f1e22"
Nov 03 18:33:06.193792 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:06.193769    5315 scope.go:122] "RemoveContainer" containerID="2d38c4c3c095eda7db19aa5715f1515dcfcd2e141470016968c035230a7db175"
Nov 03 18:33:06.201381 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:06.201362    5315 scope.go:122] "RemoveContainer" containerID="7d3fdbc061fa247a304c2bb2b3036f633b3ece0b7ce29bdd90d64f9ab5611e4c"
Nov 03 18:33:11.065068 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:11.065027    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0d7aec-7vf8q"]
Nov 03 18:33:11.065498 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:11.065407    5315 cpu_manager.go:414] "containerMap: removing container" podUID="9b52a2b0-d10c-4d06-9f78-e318f6a02973" containerName="revoke-keys"
Nov 03 18:33:11.065498 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:11.065417    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b52a2b0-d10c-4d06-9f78-e318f6a02973" containerName="revoke-keys"
Nov 03 18:33:11.065498 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:11.065481    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b52a2b0-d10c-4d06-9f78-e318f6a02973" containerName="revoke-keys"
Nov 03 18:33:11.070368 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:11.070343    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0d7aec-7vf8q"
Nov 03 18:33:11.119004 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:11.118969    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:33:11.119194 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:11.119161    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:33:11.154891 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:11.154854    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfrxj\" (UniqueName: \"kubernetes.io/projected/1a3db612-caa8-4042-bda0-5b98f41cfc4e-kube-api-access-nfrxj\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0d7aec-7vf8q\" (UID: \"1a3db612-caa8-4042-bda0-5b98f41cfc4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0d7aec-7vf8q"
Nov 03 18:33:11.155330 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:11.155015    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a3db612-caa8-4042-bda0-5b98f41cfc4e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0d7aec-7vf8q\" (UID: \"1a3db612-caa8-4042-bda0-5b98f41cfc4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0d7aec-7vf8q"
Nov 03 18:33:11.155330 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:11.155054    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a3db612-caa8-4042-bda0-5b98f41cfc4e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0d7aec-7vf8q\" (UID: \"1a3db612-caa8-4042-bda0-5b98f41cfc4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0d7aec-7vf8q"
Nov 03 18:33:11.165071 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:11.165044    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:33:11.165379 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:11.165361    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:33:11.167026 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:11.167001    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0d7aec-7vf8q"]
Nov 03 18:33:11.255686 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:11.255649    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfrxj\" (UniqueName: \"kubernetes.io/projected/1a3db612-caa8-4042-bda0-5b98f41cfc4e-kube-api-access-nfrxj\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0d7aec-7vf8q\" (UID: \"1a3db612-caa8-4042-bda0-5b98f41cfc4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0d7aec-7vf8q"
Nov 03 18:33:11.255941 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:11.255788    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a3db612-caa8-4042-bda0-5b98f41cfc4e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0d7aec-7vf8q\" (UID: \"1a3db612-caa8-4042-bda0-5b98f41cfc4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0d7aec-7vf8q"
Nov 03 18:33:11.255941 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:11.255835    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a3db612-caa8-4042-bda0-5b98f41cfc4e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0d7aec-7vf8q\" (UID: \"1a3db612-caa8-4042-bda0-5b98f41cfc4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0d7aec-7vf8q"
Nov 03 18:33:11.256488 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:11.256464    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a3db612-caa8-4042-bda0-5b98f41cfc4e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0d7aec-7vf8q\" (UID: \"1a3db612-caa8-4042-bda0-5b98f41cfc4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0d7aec-7vf8q"
Nov 03 18:33:11.258147 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:11.258123    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a3db612-caa8-4042-bda0-5b98f41cfc4e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0d7aec-7vf8q\" (UID: \"1a3db612-caa8-4042-bda0-5b98f41cfc4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0d7aec-7vf8q"
Nov 03 18:33:11.312870 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:11.312835    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfrxj\" (UniqueName: \"kubernetes.io/projected/1a3db612-caa8-4042-bda0-5b98f41cfc4e-kube-api-access-nfrxj\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0d7aec-7vf8q\" (UID: \"1a3db612-caa8-4042-bda0-5b98f41cfc4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0d7aec-7vf8q"
Nov 03 18:33:11.382865 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:11.382787    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0d7aec-7vf8q"
Nov 03 18:33:11.751112 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:11.751025    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0d7aec-7vf8q"]
Nov 03 18:33:11.758057 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:33:11.758024    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a3db612_caa8_4042_bda0_5b98f41cfc4e.slice/crio-7b9c6883d72db5a829457b4a375feea29d89f49f5b0c5d36cad00d39e0bc925e WatchSource:0}: Error finding container 7b9c6883d72db5a829457b4a375feea29d89f49f5b0c5d36cad00d39e0bc925e: Status 404 returned error can't find the container with id 7b9c6883d72db5a829457b4a375feea29d89f49f5b0c5d36cad00d39e0bc925e
Nov 03 18:33:12.598982 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:12.598229    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0d7aec-7vf8q" event={"ID":"1a3db612-caa8-4042-bda0-5b98f41cfc4e","Type":"ContainerStarted","Data":"d6c2d8f0abbe6cd01ccc52b1083465e59413b6b765a05c1ae4b226ce3861c51b"}
Nov 03 18:33:12.598982 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:12.598300    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0d7aec-7vf8q" event={"ID":"1a3db612-caa8-4042-bda0-5b98f41cfc4e","Type":"ContainerStarted","Data":"7b9c6883d72db5a829457b4a375feea29d89f49f5b0c5d36cad00d39e0bc925e"}
Nov 03 18:33:12.706892 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:12.706840    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0d7aec-7vf8q" podStartSLOduration=1.706825395 podStartE2EDuration="1.706825395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:33:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:33:12.706665291 +0000 UTC m=+2831.722863361" watchObservedRunningTime="2026-11-03 18:33:12.706825395 +0000 UTC m=+2831.723023463"
Nov 03 18:33:13.604588 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:13.604544    5315 generic.go:356] "Generic (PLEG): container finished" podID="1a3db612-caa8-4042-bda0-5b98f41cfc4e" containerID="d6c2d8f0abbe6cd01ccc52b1083465e59413b6b765a05c1ae4b226ce3861c51b" exitCode=0
Nov 03 18:33:13.605035 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:13.604674    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0d7aec-7vf8q" event={"ID":"1a3db612-caa8-4042-bda0-5b98f41cfc4e","Type":"ContainerDied","Data":"d6c2d8f0abbe6cd01ccc52b1083465e59413b6b765a05c1ae4b226ce3861c51b"}
Nov 03 18:33:14.744923 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:14.744896    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0d7aec-7vf8q"
Nov 03 18:33:14.891593 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:14.891509    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a3db612-caa8-4042-bda0-5b98f41cfc4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a3db612-caa8-4042-bda0-5b98f41cfc4e-maas-api-openshift-ca\") pod \"1a3db612-caa8-4042-bda0-5b98f41cfc4e\" (UID: \"1a3db612-caa8-4042-bda0-5b98f41cfc4e\") "
Nov 03 18:33:14.891778 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:14.891603    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a3db612-caa8-4042-bda0-5b98f41cfc4e-kube-api-access-nfrxj\" (UniqueName: \"kubernetes.io/projected/1a3db612-caa8-4042-bda0-5b98f41cfc4e-kube-api-access-nfrxj\") pod \"1a3db612-caa8-4042-bda0-5b98f41cfc4e\" (UID: \"1a3db612-caa8-4042-bda0-5b98f41cfc4e\") "
Nov 03 18:33:14.891778 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:14.891675    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a3db612-caa8-4042-bda0-5b98f41cfc4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a3db612-caa8-4042-bda0-5b98f41cfc4e-maas-api-xks-ca\") pod \"1a3db612-caa8-4042-bda0-5b98f41cfc4e\" (UID: \"1a3db612-caa8-4042-bda0-5b98f41cfc4e\") "
Nov 03 18:33:14.891981 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:14.891952    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a3db612-caa8-4042-bda0-5b98f41cfc4e-maas-api-openshift-ca" pod "1a3db612-caa8-4042-bda0-5b98f41cfc4e" (UID: "1a3db612-caa8-4042-bda0-5b98f41cfc4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:33:14.893799 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:14.893773    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a3db612-caa8-4042-bda0-5b98f41cfc4e-kube-api-access-nfrxj" pod "1a3db612-caa8-4042-bda0-5b98f41cfc4e" (UID: "1a3db612-caa8-4042-bda0-5b98f41cfc4e"). InnerVolumeSpecName "kube-api-access-nfrxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:33:14.893927 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:14.893890    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a3db612-caa8-4042-bda0-5b98f41cfc4e-maas-api-xks-ca" pod "1a3db612-caa8-4042-bda0-5b98f41cfc4e" (UID: "1a3db612-caa8-4042-bda0-5b98f41cfc4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:33:14.992389 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:14.992352    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a3db612-caa8-4042-bda0-5b98f41cfc4e-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:33:14.992389 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:14.992383    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfrxj\" (UniqueName: \"kubernetes.io/projected/1a3db612-caa8-4042-bda0-5b98f41cfc4e-kube-api-access-nfrxj\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:33:14.992389 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:14.992395    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a3db612-caa8-4042-bda0-5b98f41cfc4e-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:33:15.470443 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:15.470405    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-0d7aec/simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z"]
Nov 03 18:33:15.614744 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:15.614711    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0d7aec-7vf8q"
Nov 03 18:33:15.614925 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:15.614710    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0d7aec-7vf8q" event={"ID":"1a3db612-caa8-4042-bda0-5b98f41cfc4e","Type":"ContainerDied","Data":"7b9c6883d72db5a829457b4a375feea29d89f49f5b0c5d36cad00d39e0bc925e"}
Nov 03 18:33:15.614925 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:15.614844    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b9c6883d72db5a829457b4a375feea29d89f49f5b0c5d36cad00d39e0bc925e"
Nov 03 18:33:24.576447 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:33:24.576410    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:33:29.044374 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:29.044339    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f68c122-mct4t"]
Nov 03 18:33:29.046849 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:29.046827    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f68c122-mct4t"]
Nov 03 18:33:29.107270 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:29.107229    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb" path="/var/lib/kubelet/pods/bd908b86-7972-4b0a-8ce1-e61fc0ae1fdb/volumes"
Nov 03 18:33:30.490542 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:30.490506    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-0d7aec/simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z" podUID="a1eed39a-f722-4b9e-8c36-c591b39c58d5" containerName="main" containerID="cri-o://b4dc4939680411fe305fef906bf0c7badcee0056fcdfb89b42b5683ad40463a7" gracePeriod=45
Nov 03 18:33:30.685143 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:30.685109    5315 generic.go:356] "Generic (PLEG): container finished" podID="a1eed39a-f722-4b9e-8c36-c591b39c58d5" containerID="b4dc4939680411fe305fef906bf0c7badcee0056fcdfb89b42b5683ad40463a7" exitCode=0
Nov 03 18:33:30.685314 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:30.685184    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-0d7aec/simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z" event={"ID":"a1eed39a-f722-4b9e-8c36-c591b39c58d5","Type":"ContainerDied","Data":"b4dc4939680411fe305fef906bf0c7badcee0056fcdfb89b42b5683ad40463a7"}
Nov 03 18:33:30.737291 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:30.737262    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-0d7aec/simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z"
Nov 03 18:33:30.849032 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:30.848993    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1eed39a-f722-4b9e-8c36-c591b39c58d5-home\" (UniqueName: \"kubernetes.io/empty-dir/a1eed39a-f722-4b9e-8c36-c591b39c58d5-home\") pod \"a1eed39a-f722-4b9e-8c36-c591b39c58d5\" (UID: \"a1eed39a-f722-4b9e-8c36-c591b39c58d5\") "
Nov 03 18:33:30.849385 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:30.849362    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1eed39a-f722-4b9e-8c36-c591b39c58d5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1eed39a-f722-4b9e-8c36-c591b39c58d5-tmp-dir\") pod \"a1eed39a-f722-4b9e-8c36-c591b39c58d5\" (UID: \"a1eed39a-f722-4b9e-8c36-c591b39c58d5\") "
Nov 03 18:33:30.849556 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:30.849538    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1eed39a-f722-4b9e-8c36-c591b39c58d5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1eed39a-f722-4b9e-8c36-c591b39c58d5-model-cache\") pod \"a1eed39a-f722-4b9e-8c36-c591b39c58d5\" (UID: \"a1eed39a-f722-4b9e-8c36-c591b39c58d5\") "
Nov 03 18:33:30.850033 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:30.850013    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1eed39a-f722-4b9e-8c36-c591b39c58d5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1eed39a-f722-4b9e-8c36-c591b39c58d5-dshm\") pod \"a1eed39a-f722-4b9e-8c36-c591b39c58d5\" (UID: \"a1eed39a-f722-4b9e-8c36-c591b39c58d5\") "
Nov 03 18:33:30.850215 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:30.850196    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1eed39a-f722-4b9e-8c36-c591b39c58d5-tls-certs\" (UniqueName: \"kubernetes.io/secret/a1eed39a-f722-4b9e-8c36-c591b39c58d5-tls-certs\") pod \"a1eed39a-f722-4b9e-8c36-c591b39c58d5\" (UID: \"a1eed39a-f722-4b9e-8c36-c591b39c58d5\") "
Nov 03 18:33:30.850347 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:30.850328    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1eed39a-f722-4b9e-8c36-c591b39c58d5-kube-api-access-v2q4s\" (UniqueName: \"kubernetes.io/projected/a1eed39a-f722-4b9e-8c36-c591b39c58d5-kube-api-access-v2q4s\") pod \"a1eed39a-f722-4b9e-8c36-c591b39c58d5\" (UID: \"a1eed39a-f722-4b9e-8c36-c591b39c58d5\") "
Nov 03 18:33:30.851238 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:30.851211    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1eed39a-f722-4b9e-8c36-c591b39c58d5-model-cache" pod "a1eed39a-f722-4b9e-8c36-c591b39c58d5" (UID: "a1eed39a-f722-4b9e-8c36-c591b39c58d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:33:30.851649 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:30.851406    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1eed39a-f722-4b9e-8c36-c591b39c58d5-home" pod "a1eed39a-f722-4b9e-8c36-c591b39c58d5" (UID: "a1eed39a-f722-4b9e-8c36-c591b39c58d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:33:30.851649 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:30.851423    5315 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1eed39a-f722-4b9e-8c36-c591b39c58d5-model-cache\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:33:30.859910 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:30.856279    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1eed39a-f722-4b9e-8c36-c591b39c58d5-dshm" pod "a1eed39a-f722-4b9e-8c36-c591b39c58d5" (UID: "a1eed39a-f722-4b9e-8c36-c591b39c58d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:33:30.859910 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:30.859064    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1eed39a-f722-4b9e-8c36-c591b39c58d5-tmp-dir" pod "a1eed39a-f722-4b9e-8c36-c591b39c58d5" (UID: "a1eed39a-f722-4b9e-8c36-c591b39c58d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:33:30.859910 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:30.859686    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1eed39a-f722-4b9e-8c36-c591b39c58d5-kube-api-access-v2q4s" pod "a1eed39a-f722-4b9e-8c36-c591b39c58d5" (UID: "a1eed39a-f722-4b9e-8c36-c591b39c58d5"). InnerVolumeSpecName "kube-api-access-v2q4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:33:30.859910 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:30.859744    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1eed39a-f722-4b9e-8c36-c591b39c58d5-tls-certs" pod "a1eed39a-f722-4b9e-8c36-c591b39c58d5" (UID: "a1eed39a-f722-4b9e-8c36-c591b39c58d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:33:30.952455 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:30.952421    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2q4s\" (UniqueName: \"kubernetes.io/projected/a1eed39a-f722-4b9e-8c36-c591b39c58d5-kube-api-access-v2q4s\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:33:30.952455 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:30.952453    5315 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1eed39a-f722-4b9e-8c36-c591b39c58d5-home\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:33:30.952455 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:30.952462    5315 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1eed39a-f722-4b9e-8c36-c591b39c58d5-tmp-dir\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:33:30.952723 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:30.952471    5315 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1eed39a-f722-4b9e-8c36-c591b39c58d5-dshm\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:33:30.952723 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:30.952479    5315 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1eed39a-f722-4b9e-8c36-c591b39c58d5-tls-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:33:31.136077 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:31.136040    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0d7aec-7vf8q"]
Nov 03 18:33:31.138415 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:31.138385    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0d7aec-7vf8q"]
Nov 03 18:33:31.692542 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:31.692504    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-0d7aec/simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z" event={"ID":"a1eed39a-f722-4b9e-8c36-c591b39c58d5","Type":"ContainerDied","Data":"afa5da9179fe5e992f6aec9915e418eec8dcfab8a20f45fe97eed6ad1bc2faad"}
Nov 03 18:33:31.692542 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:31.692526    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-0d7aec/simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z"
Nov 03 18:33:31.693149 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:31.692558    5315 scope.go:122] "RemoveContainer" containerID="b4dc4939680411fe305fef906bf0c7badcee0056fcdfb89b42b5683ad40463a7"
Nov 03 18:33:31.856066 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:31.856030    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-0d7aec/simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z"]
Nov 03 18:33:31.880790 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:31.880749    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-0d7aec/simulator-single-main-0d7aec-kserve-79587c6fb6-qgf2z"]
Nov 03 18:33:33.108485 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:33.108442    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a3db612-caa8-4042-bda0-5b98f41cfc4e" path="/var/lib/kubelet/pods/1a3db612-caa8-4042-bda0-5b98f41cfc4e/volumes"
Nov 03 18:33:33.109004 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:33.108878    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1eed39a-f722-4b9e-8c36-c591b39c58d5" path="/var/lib/kubelet/pods/a1eed39a-f722-4b9e-8c36-c591b39c58d5/volumes"
Nov 03 18:33:38.000913 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:38.000872    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62"]
Nov 03 18:33:38.001408 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:38.001145    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62" podUID="49e30524-b5c2-4986-a24a-40d31a741085" containerName="istio-proxy" containerID="cri-o://b5f29d313ff5abb089d1376e5ce522dde1fd2c9356baa74efef493f164fe4e93" gracePeriod=30
Nov 03 18:33:43.246129 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:43.246100    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62"
Nov 03 18:33:43.368038 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:43.368000    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/49e30524-b5c2-4986-a24a-40d31a741085-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49e30524-b5c2-4986-a24a-40d31a741085-istio-podinfo\") pod \"49e30524-b5c2-4986-a24a-40d31a741085\" (UID: \"49e30524-b5c2-4986-a24a-40d31a741085\") "
Nov 03 18:33:43.368245 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:43.368050    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49e30524-b5c2-4986-a24a-40d31a741085-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49e30524-b5c2-4986-a24a-40d31a741085-istiod-ca-cert\") pod \"49e30524-b5c2-4986-a24a-40d31a741085\" (UID: \"49e30524-b5c2-4986-a24a-40d31a741085\") "
Nov 03 18:33:43.368245 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:43.368087    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49e30524-b5c2-4986-a24a-40d31a741085-istio-token\" (UniqueName: \"kubernetes.io/projected/49e30524-b5c2-4986-a24a-40d31a741085-istio-token\") pod \"49e30524-b5c2-4986-a24a-40d31a741085\" (UID: \"49e30524-b5c2-4986-a24a-40d31a741085\") "
Nov 03 18:33:43.368245 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:43.368115    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49e30524-b5c2-4986-a24a-40d31a741085-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e30524-b5c2-4986-a24a-40d31a741085-credential-socket\") pod \"49e30524-b5c2-4986-a24a-40d31a741085\" (UID: \"49e30524-b5c2-4986-a24a-40d31a741085\") "
Nov 03 18:33:43.368245 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:43.368164    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49e30524-b5c2-4986-a24a-40d31a741085-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e30524-b5c2-4986-a24a-40d31a741085-workload-socket\") pod \"49e30524-b5c2-4986-a24a-40d31a741085\" (UID: \"49e30524-b5c2-4986-a24a-40d31a741085\") "
Nov 03 18:33:43.368245 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:43.368198    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49e30524-b5c2-4986-a24a-40d31a741085-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49e30524-b5c2-4986-a24a-40d31a741085-istio-envoy\") pod \"49e30524-b5c2-4986-a24a-40d31a741085\" (UID: \"49e30524-b5c2-4986-a24a-40d31a741085\") "
Nov 03 18:33:43.368551 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:43.368254    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49e30524-b5c2-4986-a24a-40d31a741085-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49e30524-b5c2-4986-a24a-40d31a741085-istio-data\") pod \"49e30524-b5c2-4986-a24a-40d31a741085\" (UID: \"49e30524-b5c2-4986-a24a-40d31a741085\") "
Nov 03 18:33:43.368551 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:43.368298    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49e30524-b5c2-4986-a24a-40d31a741085-kube-api-access-szkwg\" (UniqueName: \"kubernetes.io/projected/49e30524-b5c2-4986-a24a-40d31a741085-kube-api-access-szkwg\") pod \"49e30524-b5c2-4986-a24a-40d31a741085\" (UID: \"49e30524-b5c2-4986-a24a-40d31a741085\") "
Nov 03 18:33:43.368551 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:43.368408    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49e30524-b5c2-4986-a24a-40d31a741085-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49e30524-b5c2-4986-a24a-40d31a741085-workload-certs\") pod \"49e30524-b5c2-4986-a24a-40d31a741085\" (UID: \"49e30524-b5c2-4986-a24a-40d31a741085\") "
Nov 03 18:33:43.368811 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:43.368677    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49e30524-b5c2-4986-a24a-40d31a741085-istiod-ca-cert" pod "49e30524-b5c2-4986-a24a-40d31a741085" (UID: "49e30524-b5c2-4986-a24a-40d31a741085"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:33:43.368932 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:43.368891    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49e30524-b5c2-4986-a24a-40d31a741085-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:33:43.369085 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:43.369022    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49e30524-b5c2-4986-a24a-40d31a741085-workload-certs" pod "49e30524-b5c2-4986-a24a-40d31a741085" (UID: "49e30524-b5c2-4986-a24a-40d31a741085"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:33:43.369085 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:43.369058    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49e30524-b5c2-4986-a24a-40d31a741085-workload-socket" pod "49e30524-b5c2-4986-a24a-40d31a741085" (UID: "49e30524-b5c2-4986-a24a-40d31a741085"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:33:43.369085 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:43.369002    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49e30524-b5c2-4986-a24a-40d31a741085-istio-data" pod "49e30524-b5c2-4986-a24a-40d31a741085" (UID: "49e30524-b5c2-4986-a24a-40d31a741085"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:33:43.369286 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:43.369211    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49e30524-b5c2-4986-a24a-40d31a741085-credential-socket" pod "49e30524-b5c2-4986-a24a-40d31a741085" (UID: "49e30524-b5c2-4986-a24a-40d31a741085"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:33:43.370742 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:43.370711    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49e30524-b5c2-4986-a24a-40d31a741085-istio-podinfo" pod "49e30524-b5c2-4986-a24a-40d31a741085" (UID: "49e30524-b5c2-4986-a24a-40d31a741085"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:33:43.370742 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:43.370735    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49e30524-b5c2-4986-a24a-40d31a741085-kube-api-access-szkwg" pod "49e30524-b5c2-4986-a24a-40d31a741085" (UID: "49e30524-b5c2-4986-a24a-40d31a741085"). InnerVolumeSpecName "kube-api-access-szkwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:33:43.371019 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:43.371001    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49e30524-b5c2-4986-a24a-40d31a741085-istio-envoy" pod "49e30524-b5c2-4986-a24a-40d31a741085" (UID: "49e30524-b5c2-4986-a24a-40d31a741085"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:33:43.371201 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:43.371175    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49e30524-b5c2-4986-a24a-40d31a741085-istio-token" pod "49e30524-b5c2-4986-a24a-40d31a741085" (UID: "49e30524-b5c2-4986-a24a-40d31a741085"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:33:43.469804 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:43.469765    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49e30524-b5c2-4986-a24a-40d31a741085-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:33:43.469804 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:43.469796    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49e30524-b5c2-4986-a24a-40d31a741085-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:33:43.469804 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:43.469807    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49e30524-b5c2-4986-a24a-40d31a741085-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:33:43.470068 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:43.469816    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e30524-b5c2-4986-a24a-40d31a741085-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:33:43.470068 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:43.469825    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e30524-b5c2-4986-a24a-40d31a741085-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:33:43.470068 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:43.469834    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49e30524-b5c2-4986-a24a-40d31a741085-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:33:43.470068 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:43.469843    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49e30524-b5c2-4986-a24a-40d31a741085-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:33:43.470068 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:43.469852    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szkwg\" (UniqueName: \"kubernetes.io/projected/49e30524-b5c2-4986-a24a-40d31a741085-kube-api-access-szkwg\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:33:43.754731 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:43.754636    5315 generic.go:356] "Generic (PLEG): container finished" podID="49e30524-b5c2-4986-a24a-40d31a741085" containerID="b5f29d313ff5abb089d1376e5ce522dde1fd2c9356baa74efef493f164fe4e93" exitCode=0
Nov 03 18:33:43.754731 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:43.754690    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62"
Nov 03 18:33:43.754731 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:43.754714    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62" event={"ID":"49e30524-b5c2-4986-a24a-40d31a741085","Type":"ContainerDied","Data":"b5f29d313ff5abb089d1376e5ce522dde1fd2c9356baa74efef493f164fe4e93"}
Nov 03 18:33:43.754993 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:43.754773    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62" event={"ID":"49e30524-b5c2-4986-a24a-40d31a741085","Type":"ContainerDied","Data":"d79f237aa1cf6017e75c22f00c9281ff5786f62bc9a28dc2082a760454729db5"}
Nov 03 18:33:43.754993 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:43.754796    5315 scope.go:122] "RemoveContainer" containerID="b5f29d313ff5abb089d1376e5ce522dde1fd2c9356baa74efef493f164fe4e93"
Nov 03 18:33:43.763951 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:43.763933    5315 scope.go:122] "RemoveContainer" containerID="7e060bd643a568e827e7b8510392cc1467e46d84c41d4e00ef27994f8725371e"
Nov 03 18:33:43.771864 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:43.771845    5315 scope.go:122] "RemoveContainer" containerID="b5f29d313ff5abb089d1376e5ce522dde1fd2c9356baa74efef493f164fe4e93"
Nov 03 18:33:43.772114 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:33:43.772095    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5f29d313ff5abb089d1376e5ce522dde1fd2c9356baa74efef493f164fe4e93\": container with ID starting with b5f29d313ff5abb089d1376e5ce522dde1fd2c9356baa74efef493f164fe4e93 not found: ID does not exist" containerID="b5f29d313ff5abb089d1376e5ce522dde1fd2c9356baa74efef493f164fe4e93"
Nov 03 18:33:43.772165 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:43.772125    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5f29d313ff5abb089d1376e5ce522dde1fd2c9356baa74efef493f164fe4e93"} err="failed to get container status \"b5f29d313ff5abb089d1376e5ce522dde1fd2c9356baa74efef493f164fe4e93\": rpc error: code = NotFound desc = could not find container \"b5f29d313ff5abb089d1376e5ce522dde1fd2c9356baa74efef493f164fe4e93\": container with ID starting with b5f29d313ff5abb089d1376e5ce522dde1fd2c9356baa74efef493f164fe4e93 not found: ID does not exist"
Nov 03 18:33:43.772165 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:43.772147    5315 scope.go:122] "RemoveContainer" containerID="7e060bd643a568e827e7b8510392cc1467e46d84c41d4e00ef27994f8725371e"
Nov 03 18:33:43.772403 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:33:43.772385    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e060bd643a568e827e7b8510392cc1467e46d84c41d4e00ef27994f8725371e\": container with ID starting with 7e060bd643a568e827e7b8510392cc1467e46d84c41d4e00ef27994f8725371e not found: ID does not exist" containerID="7e060bd643a568e827e7b8510392cc1467e46d84c41d4e00ef27994f8725371e"
Nov 03 18:33:43.772444 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:43.772411    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e060bd643a568e827e7b8510392cc1467e46d84c41d4e00ef27994f8725371e"} err="failed to get container status \"7e060bd643a568e827e7b8510392cc1467e46d84c41d4e00ef27994f8725371e\": rpc error: code = NotFound desc = could not find container \"7e060bd643a568e827e7b8510392cc1467e46d84c41d4e00ef27994f8725371e\": container with ID starting with 7e060bd643a568e827e7b8510392cc1467e46d84c41d4e00ef27994f8725371e not found: ID does not exist"
Nov 03 18:33:43.908827 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:43.908789    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62"]
Nov 03 18:33:43.912729 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:43.912701    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-0d7aec-openshift-default-7579dd4c5c4xs62"]
Nov 03 18:33:45.103073 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:33:45.103036    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49e30524-b5c2-4986-a24a-40d31a741085" path="/var/lib/kubelet/pods/49e30524-b5c2-4986-a24a-40d31a741085/volumes"
Nov 03 18:33:54.577924 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:33:54.577883    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:34:10.044977 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:10.044943    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zg4ns"]
Nov 03 18:34:10.090812 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:10.090771    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zg4ns"]
Nov 03 18:34:11.102462 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:11.102424    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3961efda-ba14-40d9-88c4-74d4d4882900" path="/var/lib/kubelet/pods/3961efda-ba14-40d9-88c4-74d4d4882900/volumes"
Nov 03 18:34:24.576761 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:34:24.576722    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:34:25.357313 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:25.357280    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj"]
Nov 03 18:34:25.357677 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:25.357658    5315 cpu_manager.go:414] "containerMap: removing container" podUID="49e30524-b5c2-4986-a24a-40d31a741085" containerName="istio-proxy"
Nov 03 18:34:25.357735 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:25.357676    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49e30524-b5c2-4986-a24a-40d31a741085" containerName="istio-proxy"
Nov 03 18:34:25.357735 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:25.357699    5315 cpu_manager.go:414] "containerMap: removing container" podUID="1a3db612-caa8-4042-bda0-5b98f41cfc4e" containerName="revoke-keys"
Nov 03 18:34:25.357735 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:25.357704    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a3db612-caa8-4042-bda0-5b98f41cfc4e" containerName="revoke-keys"
Nov 03 18:34:25.357735 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:25.357715    5315 cpu_manager.go:414] "containerMap: removing container" podUID="49e30524-b5c2-4986-a24a-40d31a741085" containerName="istio-proxy"
Nov 03 18:34:25.357735 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:25.357721    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49e30524-b5c2-4986-a24a-40d31a741085" containerName="istio-proxy"
Nov 03 18:34:25.357901 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:25.357738    5315 cpu_manager.go:414] "containerMap: removing container" podUID="a1eed39a-f722-4b9e-8c36-c591b39c58d5" containerName="main"
Nov 03 18:34:25.357901 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:25.357746    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1eed39a-f722-4b9e-8c36-c591b39c58d5" containerName="main"
Nov 03 18:34:25.357901 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:25.357798    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1eed39a-f722-4b9e-8c36-c591b39c58d5" containerName="main"
Nov 03 18:34:25.357901 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:25.357807    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="49e30524-b5c2-4986-a24a-40d31a741085" containerName="istio-proxy"
Nov 03 18:34:25.357901 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:25.357817    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a3db612-caa8-4042-bda0-5b98f41cfc4e" containerName="revoke-keys"
Nov 03 18:34:25.362390 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:25.362365    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj"
Nov 03 18:34:25.421539 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:25.421504    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d26f8cb9-dbfe-4cac-bd52-19be39f8b31e-model-cache\") pod \"e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj\" (UID: \"d26f8cb9-dbfe-4cac-bd52-19be39f8b31e\") " pod="llm/e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj"
Nov 03 18:34:25.421731 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:25.421565    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d26f8cb9-dbfe-4cac-bd52-19be39f8b31e-tls-certs\") pod \"e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj\" (UID: \"d26f8cb9-dbfe-4cac-bd52-19be39f8b31e\") " pod="llm/e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj"
Nov 03 18:34:25.421731 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:25.421642    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d26f8cb9-dbfe-4cac-bd52-19be39f8b31e-tmp-dir\") pod \"e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj\" (UID: \"d26f8cb9-dbfe-4cac-bd52-19be39f8b31e\") " pod="llm/e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj"
Nov 03 18:34:25.421731 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:25.421685    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d26f8cb9-dbfe-4cac-bd52-19be39f8b31e-home\") pod \"e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj\" (UID: \"d26f8cb9-dbfe-4cac-bd52-19be39f8b31e\") " pod="llm/e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj"
Nov 03 18:34:25.421853 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:25.421745    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzr4x\" (UniqueName: \"kubernetes.io/projected/d26f8cb9-dbfe-4cac-bd52-19be39f8b31e-kube-api-access-rzr4x\") pod \"e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj\" (UID: \"d26f8cb9-dbfe-4cac-bd52-19be39f8b31e\") " pod="llm/e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj"
Nov 03 18:34:25.421853 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:25.421792    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d26f8cb9-dbfe-4cac-bd52-19be39f8b31e-dshm\") pod \"e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj\" (UID: \"d26f8cb9-dbfe-4cac-bd52-19be39f8b31e\") " pod="llm/e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj"
Nov 03 18:34:25.456987 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:25.456942    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-a3447aea-kserve-self-signed-certs\""
Nov 03 18:34:25.458333 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:25.458309    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj"]
Nov 03 18:34:25.522558 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:25.522520    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d26f8cb9-dbfe-4cac-bd52-19be39f8b31e-home\") pod \"e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj\" (UID: \"d26f8cb9-dbfe-4cac-bd52-19be39f8b31e\") " pod="llm/e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj"
Nov 03 18:34:25.522793 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:25.522574    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzr4x\" (UniqueName: \"kubernetes.io/projected/d26f8cb9-dbfe-4cac-bd52-19be39f8b31e-kube-api-access-rzr4x\") pod \"e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj\" (UID: \"d26f8cb9-dbfe-4cac-bd52-19be39f8b31e\") " pod="llm/e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj"
Nov 03 18:34:25.522793 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:25.522647    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d26f8cb9-dbfe-4cac-bd52-19be39f8b31e-dshm\") pod \"e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj\" (UID: \"d26f8cb9-dbfe-4cac-bd52-19be39f8b31e\") " pod="llm/e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj"
Nov 03 18:34:25.522793 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:25.522691    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d26f8cb9-dbfe-4cac-bd52-19be39f8b31e-model-cache\") pod \"e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj\" (UID: \"d26f8cb9-dbfe-4cac-bd52-19be39f8b31e\") " pod="llm/e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj"
Nov 03 18:34:25.522793 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:25.522764    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d26f8cb9-dbfe-4cac-bd52-19be39f8b31e-tls-certs\") pod \"e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj\" (UID: \"d26f8cb9-dbfe-4cac-bd52-19be39f8b31e\") " pod="llm/e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj"
Nov 03 18:34:25.522793 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:25.522793    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d26f8cb9-dbfe-4cac-bd52-19be39f8b31e-tmp-dir\") pod \"e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj\" (UID: \"d26f8cb9-dbfe-4cac-bd52-19be39f8b31e\") " pod="llm/e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj"
Nov 03 18:34:25.523139 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:25.523114    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d26f8cb9-dbfe-4cac-bd52-19be39f8b31e-home\") pod \"e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj\" (UID: \"d26f8cb9-dbfe-4cac-bd52-19be39f8b31e\") " pod="llm/e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj"
Nov 03 18:34:25.523218 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:25.523148    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d26f8cb9-dbfe-4cac-bd52-19be39f8b31e-tmp-dir\") pod \"e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj\" (UID: \"d26f8cb9-dbfe-4cac-bd52-19be39f8b31e\") " pod="llm/e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj"
Nov 03 18:34:25.524113 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:25.524085    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d26f8cb9-dbfe-4cac-bd52-19be39f8b31e-model-cache\") pod \"e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj\" (UID: \"d26f8cb9-dbfe-4cac-bd52-19be39f8b31e\") " pod="llm/e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj"
Nov 03 18:34:25.528457 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:25.528428    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d26f8cb9-dbfe-4cac-bd52-19be39f8b31e-dshm\") pod \"e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj\" (UID: \"d26f8cb9-dbfe-4cac-bd52-19be39f8b31e\") " pod="llm/e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj"
Nov 03 18:34:25.530557 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:25.530532    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d26f8cb9-dbfe-4cac-bd52-19be39f8b31e-tls-certs\") pod \"e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj\" (UID: \"d26f8cb9-dbfe-4cac-bd52-19be39f8b31e\") " pod="llm/e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj"
Nov 03 18:34:25.580696 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:25.580664    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzr4x\" (UniqueName: \"kubernetes.io/projected/d26f8cb9-dbfe-4cac-bd52-19be39f8b31e-kube-api-access-rzr4x\") pod \"e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj\" (UID: \"d26f8cb9-dbfe-4cac-bd52-19be39f8b31e\") " pod="llm/e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj"
Nov 03 18:34:25.674675 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:25.674568    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj"
Nov 03 18:34:26.072466 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:26.072430    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj"]
Nov 03 18:34:26.079690 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:34:26.079659    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd26f8cb9_dbfe_4cac_bd52_19be39f8b31e.slice/crio-47ca0c49b2055c64c00785c8fcef9bddf935dfa0ad5195e20e40b10775823cc3 WatchSource:0}: Error finding container 47ca0c49b2055c64c00785c8fcef9bddf935dfa0ad5195e20e40b10775823cc3: Status 404 returned error can't find the container with id 47ca0c49b2055c64c00785c8fcef9bddf935dfa0ad5195e20e40b10775823cc3
Nov 03 18:34:27.002200 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:27.002163    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj" event={"ID":"d26f8cb9-dbfe-4cac-bd52-19be39f8b31e","Type":"ContainerStarted","Data":"356bc7c0c1ce6a219c1a34965d18d04f0cc52575301941837d256502e532876e"}
Nov 03 18:34:27.002640 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:27.002208    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj" event={"ID":"d26f8cb9-dbfe-4cac-bd52-19be39f8b31e","Type":"ContainerStarted","Data":"47ca0c49b2055c64c00785c8fcef9bddf935dfa0ad5195e20e40b10775823cc3"}
Nov 03 18:34:27.113795 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:27.113738    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj" podStartSLOduration=2.113725863 podStartE2EDuration="2.113725863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:34:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:34:27.112862678 +0000 UTC m=+2906.129060748" watchObservedRunningTime="2026-11-03 18:34:27.113725863 +0000 UTC m=+2906.129923931"
Nov 03 18:34:27.675578 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:27.675527    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj"
Nov 03 18:34:35.675777 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:35.675719    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj"
Nov 03 18:34:35.688219 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:35.688189    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj"
Nov 03 18:34:36.071808 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:36.071774    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj"
Nov 03 18:34:46.725458 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:46.725413    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-vz2p7"]
Nov 03 18:34:46.726136 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:46.726078    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="49e30524-b5c2-4986-a24a-40d31a741085" containerName="istio-proxy"
Nov 03 18:34:46.729262 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:46.729240    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-vz2p7"
Nov 03 18:34:46.777791 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:46.777754    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:34:46.823736 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:46.823705    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:34:46.823924 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:46.823785    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:34:46.823924 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:46.823845    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:34:46.825245 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:46.825225    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-vz2p7"]
Nov 03 18:34:46.871573 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:46.871537    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb2jd\" (UniqueName: \"kubernetes.io/projected/bf6a4396-a6ee-4756-a602-a15156edc081-kube-api-access-qb2jd\") pod \"maas-api-revoke-sub-452b6d7b-vz2p7\" (UID: \"bf6a4396-a6ee-4756-a602-a15156edc081\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-vz2p7"
Nov 03 18:34:46.871774 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:46.871589    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf6a4396-a6ee-4756-a602-a15156edc081-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-vz2p7\" (UID: \"bf6a4396-a6ee-4756-a602-a15156edc081\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-vz2p7"
Nov 03 18:34:46.871774 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:46.871688    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf6a4396-a6ee-4756-a602-a15156edc081-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-vz2p7\" (UID: \"bf6a4396-a6ee-4756-a602-a15156edc081\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-vz2p7"
Nov 03 18:34:46.972726 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:46.972688    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf6a4396-a6ee-4756-a602-a15156edc081-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-vz2p7\" (UID: \"bf6a4396-a6ee-4756-a602-a15156edc081\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-vz2p7"
Nov 03 18:34:46.972932 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:46.972739    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf6a4396-a6ee-4756-a602-a15156edc081-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-vz2p7\" (UID: \"bf6a4396-a6ee-4756-a602-a15156edc081\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-vz2p7"
Nov 03 18:34:46.972932 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:46.972857    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb2jd\" (UniqueName: \"kubernetes.io/projected/bf6a4396-a6ee-4756-a602-a15156edc081-kube-api-access-qb2jd\") pod \"maas-api-revoke-sub-452b6d7b-vz2p7\" (UID: \"bf6a4396-a6ee-4756-a602-a15156edc081\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-vz2p7"
Nov 03 18:34:46.973387 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:46.973367    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf6a4396-a6ee-4756-a602-a15156edc081-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-vz2p7\" (UID: \"bf6a4396-a6ee-4756-a602-a15156edc081\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-vz2p7"
Nov 03 18:34:46.975174 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:46.975155    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf6a4396-a6ee-4756-a602-a15156edc081-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-vz2p7\" (UID: \"bf6a4396-a6ee-4756-a602-a15156edc081\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-vz2p7"
Nov 03 18:34:47.030101 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:47.030074    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb2jd\" (UniqueName: \"kubernetes.io/projected/bf6a4396-a6ee-4756-a602-a15156edc081-kube-api-access-qb2jd\") pod \"maas-api-revoke-sub-452b6d7b-vz2p7\" (UID: \"bf6a4396-a6ee-4756-a602-a15156edc081\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-vz2p7"
Nov 03 18:34:47.042966 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:47.042937    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-vz2p7"
Nov 03 18:34:47.411154 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:47.411067    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-vz2p7"]
Nov 03 18:34:47.418109 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:34:47.418075    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf6a4396_a6ee_4756_a602_a15156edc081.slice/crio-8eaed8a933da604b31cd3f13ea829cc90ae6f4413a96191d1bf8661cc00607f2 WatchSource:0}: Error finding container 8eaed8a933da604b31cd3f13ea829cc90ae6f4413a96191d1bf8661cc00607f2: Status 404 returned error can't find the container with id 8eaed8a933da604b31cd3f13ea829cc90ae6f4413a96191d1bf8661cc00607f2
Nov 03 18:34:48.130668 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:48.130634    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-vz2p7" event={"ID":"bf6a4396-a6ee-4756-a602-a15156edc081","Type":"ContainerStarted","Data":"28e44f806d53f48ad6a33464f2e76a86ff5ea674eaaafb5f49ec5e59e322f4b8"}
Nov 03 18:34:48.131066 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:48.130689    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-vz2p7" event={"ID":"bf6a4396-a6ee-4756-a602-a15156edc081","Type":"ContainerStarted","Data":"8eaed8a933da604b31cd3f13ea829cc90ae6f4413a96191d1bf8661cc00607f2"}
Nov 03 18:34:48.238423 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:48.238368    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-vz2p7" podStartSLOduration=2.238355406 podStartE2EDuration="2.238355406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:34:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:34:48.237600231 +0000 UTC m=+2927.253798299" watchObservedRunningTime="2026-11-03 18:34:48.238355406 +0000 UTC m=+2927.254553472"
Nov 03 18:34:49.139916 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:49.139872    5315 generic.go:356] "Generic (PLEG): container finished" podID="bf6a4396-a6ee-4756-a602-a15156edc081" containerID="28e44f806d53f48ad6a33464f2e76a86ff5ea674eaaafb5f49ec5e59e322f4b8" exitCode=0
Nov 03 18:34:49.140428 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:49.139950    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-vz2p7" event={"ID":"bf6a4396-a6ee-4756-a602-a15156edc081","Type":"ContainerDied","Data":"28e44f806d53f48ad6a33464f2e76a86ff5ea674eaaafb5f49ec5e59e322f4b8"}
Nov 03 18:34:50.920558 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:50.917034    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-vz2p7"
Nov 03 18:34:50.965935 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:50.965889    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf6a4396-a6ee-4756-a602-a15156edc081-kube-api-access-qb2jd\" (UniqueName: \"kubernetes.io/projected/bf6a4396-a6ee-4756-a602-a15156edc081-kube-api-access-qb2jd\") pod \"bf6a4396-a6ee-4756-a602-a15156edc081\" (UID: \"bf6a4396-a6ee-4756-a602-a15156edc081\") "
Nov 03 18:34:50.966144 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:50.966037    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf6a4396-a6ee-4756-a602-a15156edc081-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf6a4396-a6ee-4756-a602-a15156edc081-maas-api-openshift-ca\") pod \"bf6a4396-a6ee-4756-a602-a15156edc081\" (UID: \"bf6a4396-a6ee-4756-a602-a15156edc081\") "
Nov 03 18:34:50.966144 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:50.966124    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf6a4396-a6ee-4756-a602-a15156edc081-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf6a4396-a6ee-4756-a602-a15156edc081-maas-api-xks-ca\") pod \"bf6a4396-a6ee-4756-a602-a15156edc081\" (UID: \"bf6a4396-a6ee-4756-a602-a15156edc081\") "
Nov 03 18:34:50.985635 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:50.974109    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf6a4396-a6ee-4756-a602-a15156edc081-maas-api-openshift-ca" pod "bf6a4396-a6ee-4756-a602-a15156edc081" (UID: "bf6a4396-a6ee-4756-a602-a15156edc081"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:34:51.006911 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:50.988153    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf6a4396-a6ee-4756-a602-a15156edc081-kube-api-access-qb2jd" pod "bf6a4396-a6ee-4756-a602-a15156edc081" (UID: "bf6a4396-a6ee-4756-a602-a15156edc081"). InnerVolumeSpecName "kube-api-access-qb2jd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:34:51.006911 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:50.992046    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf6a4396-a6ee-4756-a602-a15156edc081-maas-api-xks-ca" pod "bf6a4396-a6ee-4756-a602-a15156edc081" (UID: "bf6a4396-a6ee-4756-a602-a15156edc081"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:34:51.075235 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:51.075191    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf6a4396-a6ee-4756-a602-a15156edc081-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:34:51.075505 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:51.075483    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb2jd\" (UniqueName: \"kubernetes.io/projected/bf6a4396-a6ee-4756-a602-a15156edc081-kube-api-access-qb2jd\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:34:51.075679 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:51.075661    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf6a4396-a6ee-4756-a602-a15156edc081-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:34:51.335634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:51.334885    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-vz2p7" event={"ID":"bf6a4396-a6ee-4756-a602-a15156edc081","Type":"ContainerDied","Data":"8eaed8a933da604b31cd3f13ea829cc90ae6f4413a96191d1bf8661cc00607f2"}
Nov 03 18:34:51.335634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:51.334970    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8eaed8a933da604b31cd3f13ea829cc90ae6f4413a96191d1bf8661cc00607f2"
Nov 03 18:34:51.335634 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:34:51.335063    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-vz2p7"
Nov 03 18:34:54.581082 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:34:54.581035    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:35:06.261632 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:06.261021    5315 scope.go:122] "RemoveContainer" containerID="ccbaa2f218c71ebc1b53ec7a6860b9883d50f782338aa8f4697ce8e22baa80de"
Nov 03 18:35:06.348372 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:06.347916    5315 scope.go:122] "RemoveContainer" containerID="6c674a95cc949138c940c9984b541fc6716191aff24f5a0ab53794ce3c972d9f"
Nov 03 18:35:17.315768 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:17.315727    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj"]
Nov 03 18:35:21.560773 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:21.560736    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-b5j78"]
Nov 03 18:35:21.561405 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:21.561277    5315 cpu_manager.go:414] "containerMap: removing container" podUID="bf6a4396-a6ee-4756-a602-a15156edc081" containerName="revoke-keys"
Nov 03 18:35:21.561405 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:21.561298    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf6a4396-a6ee-4756-a602-a15156edc081" containerName="revoke-keys"
Nov 03 18:35:21.561538 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:21.561426    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf6a4396-a6ee-4756-a602-a15156edc081" containerName="revoke-keys"
Nov 03 18:35:21.566009 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:21.565976    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-b5j78"
Nov 03 18:35:21.614594 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:21.614555    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:35:21.658146 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:21.658110    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9108b98c-1c45-49d5-9c62-3a6272b39cad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-b5j78\" (UID: \"9108b98c-1c45-49d5-9c62-3a6272b39cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-b5j78"
Nov 03 18:35:21.658329 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:21.658172    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5vnj\" (UniqueName: \"kubernetes.io/projected/9108b98c-1c45-49d5-9c62-3a6272b39cad-kube-api-access-x5vnj\") pod \"maas-api-revoke-sub-3e31a506-b5j78\" (UID: \"9108b98c-1c45-49d5-9c62-3a6272b39cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-b5j78"
Nov 03 18:35:21.658329 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:21.658201    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9108b98c-1c45-49d5-9c62-3a6272b39cad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-b5j78\" (UID: \"9108b98c-1c45-49d5-9c62-3a6272b39cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-b5j78"
Nov 03 18:35:21.660346 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:21.660314    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:35:21.660541 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:21.660395    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:35:21.660631 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:21.660415    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:35:21.661951 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:21.661928    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-b5j78"]
Nov 03 18:35:21.759502 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:21.759471    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9108b98c-1c45-49d5-9c62-3a6272b39cad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-b5j78\" (UID: \"9108b98c-1c45-49d5-9c62-3a6272b39cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-b5j78"
Nov 03 18:35:21.759723 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:21.759527    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5vnj\" (UniqueName: \"kubernetes.io/projected/9108b98c-1c45-49d5-9c62-3a6272b39cad-kube-api-access-x5vnj\") pod \"maas-api-revoke-sub-3e31a506-b5j78\" (UID: \"9108b98c-1c45-49d5-9c62-3a6272b39cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-b5j78"
Nov 03 18:35:21.759723 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:21.759560    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9108b98c-1c45-49d5-9c62-3a6272b39cad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-b5j78\" (UID: \"9108b98c-1c45-49d5-9c62-3a6272b39cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-b5j78"
Nov 03 18:35:21.760285 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:21.760265    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9108b98c-1c45-49d5-9c62-3a6272b39cad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-b5j78\" (UID: \"9108b98c-1c45-49d5-9c62-3a6272b39cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-b5j78"
Nov 03 18:35:21.761769 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:21.761742    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9108b98c-1c45-49d5-9c62-3a6272b39cad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-b5j78\" (UID: \"9108b98c-1c45-49d5-9c62-3a6272b39cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-b5j78"
Nov 03 18:35:21.816538 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:21.816465    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5vnj\" (UniqueName: \"kubernetes.io/projected/9108b98c-1c45-49d5-9c62-3a6272b39cad-kube-api-access-x5vnj\") pod \"maas-api-revoke-sub-3e31a506-b5j78\" (UID: \"9108b98c-1c45-49d5-9c62-3a6272b39cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-b5j78"
Nov 03 18:35:21.878664 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:21.878628    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-b5j78"
Nov 03 18:35:22.249303 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:22.249211    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-b5j78"]
Nov 03 18:35:22.258528 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:35:22.258494    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9108b98c_1c45_49d5_9c62_3a6272b39cad.slice/crio-3a5d8fd84ad9b93ca4699c4991c23cb4a2bad1ba58fdb8b6208ee3b112a9f95b WatchSource:0}: Error finding container 3a5d8fd84ad9b93ca4699c4991c23cb4a2bad1ba58fdb8b6208ee3b112a9f95b: Status 404 returned error can't find the container with id 3a5d8fd84ad9b93ca4699c4991c23cb4a2bad1ba58fdb8b6208ee3b112a9f95b
Nov 03 18:35:22.523277 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:22.523168    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-b5j78" event={"ID":"9108b98c-1c45-49d5-9c62-3a6272b39cad","Type":"ContainerStarted","Data":"3a5d8fd84ad9b93ca4699c4991c23cb4a2bad1ba58fdb8b6208ee3b112a9f95b"}
Nov 03 18:35:23.614694 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:23.614656    5315 generic.go:356] "Generic (PLEG): container finished" podID="9108b98c-1c45-49d5-9c62-3a6272b39cad" containerID="a5f29d82ccd75884a3e9db9e23f7392ab19e538acaeae871cb9aea7fddd91787" exitCode=0
Nov 03 18:35:23.615321 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:23.615273    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-b5j78" event={"ID":"9108b98c-1c45-49d5-9c62-3a6272b39cad","Type":"ContainerDied","Data":"a5f29d82ccd75884a3e9db9e23f7392ab19e538acaeae871cb9aea7fddd91787"}
Nov 03 18:35:24.701889 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:35:24.701842    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:35:25.019312 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:25.019281    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-b5j78"
Nov 03 18:35:25.152353 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:25.152311    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9108b98c-1c45-49d5-9c62-3a6272b39cad-kube-api-access-x5vnj\" (UniqueName: \"kubernetes.io/projected/9108b98c-1c45-49d5-9c62-3a6272b39cad-kube-api-access-x5vnj\") pod \"9108b98c-1c45-49d5-9c62-3a6272b39cad\" (UID: \"9108b98c-1c45-49d5-9c62-3a6272b39cad\") "
Nov 03 18:35:25.152528 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:25.152366    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9108b98c-1c45-49d5-9c62-3a6272b39cad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9108b98c-1c45-49d5-9c62-3a6272b39cad-maas-api-openshift-ca\") pod \"9108b98c-1c45-49d5-9c62-3a6272b39cad\" (UID: \"9108b98c-1c45-49d5-9c62-3a6272b39cad\") "
Nov 03 18:35:25.152528 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:25.152464    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9108b98c-1c45-49d5-9c62-3a6272b39cad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9108b98c-1c45-49d5-9c62-3a6272b39cad-maas-api-xks-ca\") pod \"9108b98c-1c45-49d5-9c62-3a6272b39cad\" (UID: \"9108b98c-1c45-49d5-9c62-3a6272b39cad\") "
Nov 03 18:35:25.152855 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:25.152791    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9108b98c-1c45-49d5-9c62-3a6272b39cad-maas-api-openshift-ca" pod "9108b98c-1c45-49d5-9c62-3a6272b39cad" (UID: "9108b98c-1c45-49d5-9c62-3a6272b39cad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:35:25.152975 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:25.152953    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9108b98c-1c45-49d5-9c62-3a6272b39cad-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:35:25.154577 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:25.154550    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9108b98c-1c45-49d5-9c62-3a6272b39cad-maas-api-xks-ca" pod "9108b98c-1c45-49d5-9c62-3a6272b39cad" (UID: "9108b98c-1c45-49d5-9c62-3a6272b39cad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:35:25.154702 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:25.154658    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9108b98c-1c45-49d5-9c62-3a6272b39cad-kube-api-access-x5vnj" pod "9108b98c-1c45-49d5-9c62-3a6272b39cad" (UID: "9108b98c-1c45-49d5-9c62-3a6272b39cad"). InnerVolumeSpecName "kube-api-access-x5vnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:35:25.254465 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:25.254428    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5vnj\" (UniqueName: \"kubernetes.io/projected/9108b98c-1c45-49d5-9c62-3a6272b39cad-kube-api-access-x5vnj\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:35:25.254465 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:25.254458    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9108b98c-1c45-49d5-9c62-3a6272b39cad-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:35:25.630700 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:25.630667    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-b5j78"
Nov 03 18:35:25.630700 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:25.630691    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-b5j78" event={"ID":"9108b98c-1c45-49d5-9c62-3a6272b39cad","Type":"ContainerDied","Data":"3a5d8fd84ad9b93ca4699c4991c23cb4a2bad1ba58fdb8b6208ee3b112a9f95b"}
Nov 03 18:35:25.630960 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:25.630736    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a5d8fd84ad9b93ca4699c4991c23cb4a2bad1ba58fdb8b6208ee3b112a9f95b"
Nov 03 18:35:32.336434 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:32.336380    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj" podUID="d26f8cb9-dbfe-4cac-bd52-19be39f8b31e" containerName="main" containerID="cri-o://356bc7c0c1ce6a219c1a34965d18d04f0cc52575301941837d256502e532876e" gracePeriod=45
Nov 03 18:35:32.578368 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:32.578335    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj"
Nov 03 18:35:32.663096 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:32.663006    5315 generic.go:356] "Generic (PLEG): container finished" podID="d26f8cb9-dbfe-4cac-bd52-19be39f8b31e" containerID="356bc7c0c1ce6a219c1a34965d18d04f0cc52575301941837d256502e532876e" exitCode=0
Nov 03 18:35:32.663096 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:32.663046    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj"
Nov 03 18:35:32.663096 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:32.663072    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj" event={"ID":"d26f8cb9-dbfe-4cac-bd52-19be39f8b31e","Type":"ContainerDied","Data":"356bc7c0c1ce6a219c1a34965d18d04f0cc52575301941837d256502e532876e"}
Nov 03 18:35:32.663403 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:32.663107    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj" event={"ID":"d26f8cb9-dbfe-4cac-bd52-19be39f8b31e","Type":"ContainerDied","Data":"47ca0c49b2055c64c00785c8fcef9bddf935dfa0ad5195e20e40b10775823cc3"}
Nov 03 18:35:32.663403 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:32.663123    5315 scope.go:122] "RemoveContainer" containerID="356bc7c0c1ce6a219c1a34965d18d04f0cc52575301941837d256502e532876e"
Nov 03 18:35:32.673811 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:32.673780    5315 scope.go:122] "RemoveContainer" containerID="356bc7c0c1ce6a219c1a34965d18d04f0cc52575301941837d256502e532876e"
Nov 03 18:35:32.674131 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:35:32.674111    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"356bc7c0c1ce6a219c1a34965d18d04f0cc52575301941837d256502e532876e\": container with ID starting with 356bc7c0c1ce6a219c1a34965d18d04f0cc52575301941837d256502e532876e not found: ID does not exist" containerID="356bc7c0c1ce6a219c1a34965d18d04f0cc52575301941837d256502e532876e"
Nov 03 18:35:32.674187 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:32.674142    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"356bc7c0c1ce6a219c1a34965d18d04f0cc52575301941837d256502e532876e"} err="failed to get container status \"356bc7c0c1ce6a219c1a34965d18d04f0cc52575301941837d256502e532876e\": rpc error: code = NotFound desc = could not find container \"356bc7c0c1ce6a219c1a34965d18d04f0cc52575301941837d256502e532876e\": container with ID starting with 356bc7c0c1ce6a219c1a34965d18d04f0cc52575301941837d256502e532876e not found: ID does not exist"
Nov 03 18:35:32.720027 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:32.719996    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d26f8cb9-dbfe-4cac-bd52-19be39f8b31e-home\" (UniqueName: \"kubernetes.io/empty-dir/d26f8cb9-dbfe-4cac-bd52-19be39f8b31e-home\") pod \"d26f8cb9-dbfe-4cac-bd52-19be39f8b31e\" (UID: \"d26f8cb9-dbfe-4cac-bd52-19be39f8b31e\") "
Nov 03 18:35:32.720027 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:32.720033    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d26f8cb9-dbfe-4cac-bd52-19be39f8b31e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d26f8cb9-dbfe-4cac-bd52-19be39f8b31e-tmp-dir\") pod \"d26f8cb9-dbfe-4cac-bd52-19be39f8b31e\" (UID: \"d26f8cb9-dbfe-4cac-bd52-19be39f8b31e\") "
Nov 03 18:35:32.720223 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:32.720079    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d26f8cb9-dbfe-4cac-bd52-19be39f8b31e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d26f8cb9-dbfe-4cac-bd52-19be39f8b31e-dshm\") pod \"d26f8cb9-dbfe-4cac-bd52-19be39f8b31e\" (UID: \"d26f8cb9-dbfe-4cac-bd52-19be39f8b31e\") "
Nov 03 18:35:32.720223 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:32.720190    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d26f8cb9-dbfe-4cac-bd52-19be39f8b31e-kube-api-access-rzr4x\" (UniqueName: \"kubernetes.io/projected/d26f8cb9-dbfe-4cac-bd52-19be39f8b31e-kube-api-access-rzr4x\") pod \"d26f8cb9-dbfe-4cac-bd52-19be39f8b31e\" (UID: \"d26f8cb9-dbfe-4cac-bd52-19be39f8b31e\") "
Nov 03 18:35:32.720311 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:32.720262    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d26f8cb9-dbfe-4cac-bd52-19be39f8b31e-tls-certs\" (UniqueName: \"kubernetes.io/secret/d26f8cb9-dbfe-4cac-bd52-19be39f8b31e-tls-certs\") pod \"d26f8cb9-dbfe-4cac-bd52-19be39f8b31e\" (UID: \"d26f8cb9-dbfe-4cac-bd52-19be39f8b31e\") "
Nov 03 18:35:32.720357 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:32.720284    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26f8cb9-dbfe-4cac-bd52-19be39f8b31e-home" pod "d26f8cb9-dbfe-4cac-bd52-19be39f8b31e" (UID: "d26f8cb9-dbfe-4cac-bd52-19be39f8b31e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:35:32.720357 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:32.720336    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d26f8cb9-dbfe-4cac-bd52-19be39f8b31e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d26f8cb9-dbfe-4cac-bd52-19be39f8b31e-model-cache\") pod \"d26f8cb9-dbfe-4cac-bd52-19be39f8b31e\" (UID: \"d26f8cb9-dbfe-4cac-bd52-19be39f8b31e\") "
Nov 03 18:35:32.720467 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:32.720354    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26f8cb9-dbfe-4cac-bd52-19be39f8b31e-tmp-dir" pod "d26f8cb9-dbfe-4cac-bd52-19be39f8b31e" (UID: "d26f8cb9-dbfe-4cac-bd52-19be39f8b31e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:35:32.720624 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:32.720597    5315 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d26f8cb9-dbfe-4cac-bd52-19be39f8b31e-home\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:35:32.720726 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:32.720630    5315 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d26f8cb9-dbfe-4cac-bd52-19be39f8b31e-tmp-dir\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:35:32.720726 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:32.720679    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26f8cb9-dbfe-4cac-bd52-19be39f8b31e-model-cache" pod "d26f8cb9-dbfe-4cac-bd52-19be39f8b31e" (UID: "d26f8cb9-dbfe-4cac-bd52-19be39f8b31e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:35:32.722495 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:32.722465    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d26f8cb9-dbfe-4cac-bd52-19be39f8b31e-tls-certs" pod "d26f8cb9-dbfe-4cac-bd52-19be39f8b31e" (UID: "d26f8cb9-dbfe-4cac-bd52-19be39f8b31e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:35:32.722806 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:32.722786    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26f8cb9-dbfe-4cac-bd52-19be39f8b31e-dshm" pod "d26f8cb9-dbfe-4cac-bd52-19be39f8b31e" (UID: "d26f8cb9-dbfe-4cac-bd52-19be39f8b31e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:35:32.722922 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:32.722812    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d26f8cb9-dbfe-4cac-bd52-19be39f8b31e-kube-api-access-rzr4x" pod "d26f8cb9-dbfe-4cac-bd52-19be39f8b31e" (UID: "d26f8cb9-dbfe-4cac-bd52-19be39f8b31e"). InnerVolumeSpecName "kube-api-access-rzr4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:35:32.822054 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:32.822014    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzr4x\" (UniqueName: \"kubernetes.io/projected/d26f8cb9-dbfe-4cac-bd52-19be39f8b31e-kube-api-access-rzr4x\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:35:32.822054 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:32.822048    5315 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d26f8cb9-dbfe-4cac-bd52-19be39f8b31e-tls-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:35:32.822054 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:32.822060    5315 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d26f8cb9-dbfe-4cac-bd52-19be39f8b31e-model-cache\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:35:32.822302 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:32.822069    5315 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d26f8cb9-dbfe-4cac-bd52-19be39f8b31e-dshm\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:35:33.074839 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:33.074805    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj"]
Nov 03 18:35:33.120778 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:33.120747    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-a3447aea-kserve-74bddb8c6f-gddnj"]
Nov 03 18:35:35.120633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:35.119898    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d26f8cb9-dbfe-4cac-bd52-19be39f8b31e" path="/var/lib/kubelet/pods/d26f8cb9-dbfe-4cac-bd52-19be39f8b31e/volumes"
Nov 03 18:35:52.327784 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:52.327751    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-521c9b81-jwtvw"]
Nov 03 18:35:52.328184 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:52.328107    5315 cpu_manager.go:414] "containerMap: removing container" podUID="d26f8cb9-dbfe-4cac-bd52-19be39f8b31e" containerName="main"
Nov 03 18:35:52.328184 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:52.328117    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d26f8cb9-dbfe-4cac-bd52-19be39f8b31e" containerName="main"
Nov 03 18:35:52.328184 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:52.328126    5315 cpu_manager.go:414] "containerMap: removing container" podUID="9108b98c-1c45-49d5-9c62-3a6272b39cad" containerName="revoke-keys"
Nov 03 18:35:52.328184 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:52.328132    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9108b98c-1c45-49d5-9c62-3a6272b39cad" containerName="revoke-keys"
Nov 03 18:35:52.328331 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:52.328202    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="d26f8cb9-dbfe-4cac-bd52-19be39f8b31e" containerName="main"
Nov 03 18:35:52.328331 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:52.328212    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="9108b98c-1c45-49d5-9c62-3a6272b39cad" containerName="revoke-keys"
Nov 03 18:35:52.332842 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:52.332818    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-521c9b81-jwtvw"
Nov 03 18:35:52.427364 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:52.427333    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:35:52.427364 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:52.427333    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:35:52.427573 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:52.427376    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:35:52.427573 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:52.427524    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\""
Nov 03 18:35:52.429090 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:52.429070    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-521c9b81-jwtvw"]
Nov 03 18:35:52.483947 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:52.483912    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05fd4c69-274e-4cc4-91e0-5a8a77d08705-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-521c9b81-jwtvw\" (UID: \"05fd4c69-274e-4cc4-91e0-5a8a77d08705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521c9b81-jwtvw"
Nov 03 18:35:52.484098 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:52.483961    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05fd4c69-274e-4cc4-91e0-5a8a77d08705-maas-api-xks-ca\") pod \"maas-api-revoke-sub-521c9b81-jwtvw\" (UID: \"05fd4c69-274e-4cc4-91e0-5a8a77d08705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521c9b81-jwtvw"
Nov 03 18:35:52.484098 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:52.484028    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fldt\" (UniqueName: \"kubernetes.io/projected/05fd4c69-274e-4cc4-91e0-5a8a77d08705-kube-api-access-9fldt\") pod \"maas-api-revoke-sub-521c9b81-jwtvw\" (UID: \"05fd4c69-274e-4cc4-91e0-5a8a77d08705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521c9b81-jwtvw"
Nov 03 18:35:52.585355 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:52.585264    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05fd4c69-274e-4cc4-91e0-5a8a77d08705-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-521c9b81-jwtvw\" (UID: \"05fd4c69-274e-4cc4-91e0-5a8a77d08705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521c9b81-jwtvw"
Nov 03 18:35:52.585355 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:52.585314    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05fd4c69-274e-4cc4-91e0-5a8a77d08705-maas-api-xks-ca\") pod \"maas-api-revoke-sub-521c9b81-jwtvw\" (UID: \"05fd4c69-274e-4cc4-91e0-5a8a77d08705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521c9b81-jwtvw"
Nov 03 18:35:52.585355 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:52.585330    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fldt\" (UniqueName: \"kubernetes.io/projected/05fd4c69-274e-4cc4-91e0-5a8a77d08705-kube-api-access-9fldt\") pod \"maas-api-revoke-sub-521c9b81-jwtvw\" (UID: \"05fd4c69-274e-4cc4-91e0-5a8a77d08705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521c9b81-jwtvw"
Nov 03 18:35:52.585986 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:52.585948    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05fd4c69-274e-4cc4-91e0-5a8a77d08705-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-521c9b81-jwtvw\" (UID: \"05fd4c69-274e-4cc4-91e0-5a8a77d08705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521c9b81-jwtvw"
Nov 03 18:35:52.587666 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:52.587641    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05fd4c69-274e-4cc4-91e0-5a8a77d08705-maas-api-xks-ca\") pod \"maas-api-revoke-sub-521c9b81-jwtvw\" (UID: \"05fd4c69-274e-4cc4-91e0-5a8a77d08705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521c9b81-jwtvw"
Nov 03 18:35:52.642398 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:52.642361    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fldt\" (UniqueName: \"kubernetes.io/projected/05fd4c69-274e-4cc4-91e0-5a8a77d08705-kube-api-access-9fldt\") pod \"maas-api-revoke-sub-521c9b81-jwtvw\" (UID: \"05fd4c69-274e-4cc4-91e0-5a8a77d08705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521c9b81-jwtvw"
Nov 03 18:35:52.645131 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:52.645111    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-521c9b81-jwtvw"
Nov 03 18:35:53.012846 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:53.012761    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-521c9b81-jwtvw"]
Nov 03 18:35:53.019924 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:35:53.019893    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05fd4c69_274e_4cc4_91e0_5a8a77d08705.slice/crio-5a62a0161643e63cbf473d6f304797d796aac2ff0e3f113d742000bf4b5d70e3 WatchSource:0}: Error finding container 5a62a0161643e63cbf473d6f304797d796aac2ff0e3f113d742000bf4b5d70e3: Status 404 returned error can't find the container with id 5a62a0161643e63cbf473d6f304797d796aac2ff0e3f113d742000bf4b5d70e3
Nov 03 18:35:53.824805 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:53.824758    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-521c9b81-jwtvw" event={"ID":"05fd4c69-274e-4cc4-91e0-5a8a77d08705","Type":"ContainerStarted","Data":"b2aae2ffd5e62549a98045f8b6671816f47a3202f131e03f6e3b61b245b57589"}
Nov 03 18:35:53.825256 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:53.824817    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-521c9b81-jwtvw" event={"ID":"05fd4c69-274e-4cc4-91e0-5a8a77d08705","Type":"ContainerStarted","Data":"5a62a0161643e63cbf473d6f304797d796aac2ff0e3f113d742000bf4b5d70e3"}
Nov 03 18:35:53.934269 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:53.934214    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-521c9b81-jwtvw" podStartSLOduration=1.9341998999999999 podStartE2EDuration="1.9341999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:35:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:35:53.933361292 +0000 UTC m=+2992.949559362" watchObservedRunningTime="2026-11-03 18:35:53.9341999 +0000 UTC m=+2992.950397968"
Nov 03 18:35:54.577177 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:35:54.577138    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:35:54.831105 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:54.831019    5315 generic.go:356] "Generic (PLEG): container finished" podID="05fd4c69-274e-4cc4-91e0-5a8a77d08705" containerID="b2aae2ffd5e62549a98045f8b6671816f47a3202f131e03f6e3b61b245b57589" exitCode=0
Nov 03 18:35:54.831105 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:54.831075    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-521c9b81-jwtvw" event={"ID":"05fd4c69-274e-4cc4-91e0-5a8a77d08705","Type":"ContainerDied","Data":"b2aae2ffd5e62549a98045f8b6671816f47a3202f131e03f6e3b61b245b57589"}
Nov 03 18:35:55.966453 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:55.966423    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-521c9b81-jwtvw"
Nov 03 18:35:56.122295 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:56.122213    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05fd4c69-274e-4cc4-91e0-5a8a77d08705-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05fd4c69-274e-4cc4-91e0-5a8a77d08705-maas-api-xks-ca\") pod \"05fd4c69-274e-4cc4-91e0-5a8a77d08705\" (UID: \"05fd4c69-274e-4cc4-91e0-5a8a77d08705\") "
Nov 03 18:35:56.122295 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:56.122276    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05fd4c69-274e-4cc4-91e0-5a8a77d08705-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05fd4c69-274e-4cc4-91e0-5a8a77d08705-maas-api-openshift-ca\") pod \"05fd4c69-274e-4cc4-91e0-5a8a77d08705\" (UID: \"05fd4c69-274e-4cc4-91e0-5a8a77d08705\") "
Nov 03 18:35:56.122535 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:56.122382    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05fd4c69-274e-4cc4-91e0-5a8a77d08705-kube-api-access-9fldt\" (UniqueName: \"kubernetes.io/projected/05fd4c69-274e-4cc4-91e0-5a8a77d08705-kube-api-access-9fldt\") pod \"05fd4c69-274e-4cc4-91e0-5a8a77d08705\" (UID: \"05fd4c69-274e-4cc4-91e0-5a8a77d08705\") "
Nov 03 18:35:56.122701 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:56.122675    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05fd4c69-274e-4cc4-91e0-5a8a77d08705-maas-api-openshift-ca" pod "05fd4c69-274e-4cc4-91e0-5a8a77d08705" (UID: "05fd4c69-274e-4cc4-91e0-5a8a77d08705"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:35:56.124438 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:56.124412    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05fd4c69-274e-4cc4-91e0-5a8a77d08705-kube-api-access-9fldt" pod "05fd4c69-274e-4cc4-91e0-5a8a77d08705" (UID: "05fd4c69-274e-4cc4-91e0-5a8a77d08705"). InnerVolumeSpecName "kube-api-access-9fldt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:35:56.124554 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:56.124433    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05fd4c69-274e-4cc4-91e0-5a8a77d08705-maas-api-xks-ca" pod "05fd4c69-274e-4cc4-91e0-5a8a77d08705" (UID: "05fd4c69-274e-4cc4-91e0-5a8a77d08705"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:35:56.223323 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:56.223283    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05fd4c69-274e-4cc4-91e0-5a8a77d08705-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:35:56.223323 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:56.223316    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05fd4c69-274e-4cc4-91e0-5a8a77d08705-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:35:56.223323 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:56.223325    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fldt\" (UniqueName: \"kubernetes.io/projected/05fd4c69-274e-4cc4-91e0-5a8a77d08705-kube-api-access-9fldt\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:35:56.841398 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:56.841353    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-521c9b81-jwtvw" event={"ID":"05fd4c69-274e-4cc4-91e0-5a8a77d08705","Type":"ContainerDied","Data":"5a62a0161643e63cbf473d6f304797d796aac2ff0e3f113d742000bf4b5d70e3"}
Nov 03 18:35:56.841581 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:56.841441    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a62a0161643e63cbf473d6f304797d796aac2ff0e3f113d742000bf4b5d70e3"
Nov 03 18:35:56.841581 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:35:56.841370    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-521c9b81-jwtvw"
Nov 03 18:36:01.716785 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:01.716755    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-97p65_c29d200a-c917-46e5-9cf5-21f4d952e470/ovn-acl-logging/0.log"
Nov 03 18:36:01.766142 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:01.766112    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-97p65_c29d200a-c917-46e5-9cf5-21f4d952e470/ovn-acl-logging/0.log"
Nov 03 18:36:03.043971 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:03.043934    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-lm6tb"]
Nov 03 18:36:03.089968 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:03.089933    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-lm6tb"]
Nov 03 18:36:03.102229 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:03.102198    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b92e48f-b943-4252-b487-7af6a4c8e8be" path="/var/lib/kubelet/pods/9b92e48f-b943-4252-b487-7af6a4c8e8be/volumes"
Nov 03 18:36:06.438343 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:06.438307    5315 scope.go:122] "RemoveContainer" containerID="08f6cf20e01c2687d6bf7cfc1aef9a1490577bbcfe70e6960ea0d5d999d3e629"
Nov 03 18:36:12.870725 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:12.870687    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4"]
Nov 03 18:36:12.871160 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:12.871029    5315 cpu_manager.go:414] "containerMap: removing container" podUID="05fd4c69-274e-4cc4-91e0-5a8a77d08705" containerName="revoke-keys"
Nov 03 18:36:12.871160 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:12.871040    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05fd4c69-274e-4cc4-91e0-5a8a77d08705" containerName="revoke-keys"
Nov 03 18:36:12.871160 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:12.871120    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="05fd4c69-274e-4cc4-91e0-5a8a77d08705" containerName="revoke-keys"
Nov 03 18:36:12.875664 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:12.875638    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4"
Nov 03 18:36:12.932745 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:12.932710    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-07d66926-openshift-default-dockercfg-7wsgz\""
Nov 03 18:36:12.989152 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:12.989119    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4"]
Nov 03 18:36:13.058691 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:13.058650    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5czh\" (UniqueName: \"kubernetes.io/projected/2fcbb83b-f0fc-427b-806c-62cf4fa66437-kube-api-access-r5czh\") pod \"e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4\" (UID: \"2fcbb83b-f0fc-427b-806c-62cf4fa66437\") " pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4"
Nov 03 18:36:13.058882 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:13.058698    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fcbb83b-f0fc-427b-806c-62cf4fa66437-workload-socket\") pod \"e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4\" (UID: \"2fcbb83b-f0fc-427b-806c-62cf4fa66437\") " pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4"
Nov 03 18:36:13.058882 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:13.058810    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fcbb83b-f0fc-427b-806c-62cf4fa66437-istio-podinfo\") pod \"e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4\" (UID: \"2fcbb83b-f0fc-427b-806c-62cf4fa66437\") " pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4"
Nov 03 18:36:13.058882 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:13.058863    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fcbb83b-f0fc-427b-806c-62cf4fa66437-workload-certs\") pod \"e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4\" (UID: \"2fcbb83b-f0fc-427b-806c-62cf4fa66437\") " pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4"
Nov 03 18:36:13.059036 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:13.058951    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fcbb83b-f0fc-427b-806c-62cf4fa66437-istio-token\") pod \"e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4\" (UID: \"2fcbb83b-f0fc-427b-806c-62cf4fa66437\") " pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4"
Nov 03 18:36:13.059036 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:13.058985    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fcbb83b-f0fc-427b-806c-62cf4fa66437-istio-envoy\") pod \"e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4\" (UID: \"2fcbb83b-f0fc-427b-806c-62cf4fa66437\") " pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4"
Nov 03 18:36:13.059036 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:13.059009    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fcbb83b-f0fc-427b-806c-62cf4fa66437-istiod-ca-cert\") pod \"e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4\" (UID: \"2fcbb83b-f0fc-427b-806c-62cf4fa66437\") " pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4"
Nov 03 18:36:13.059209 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:13.059129    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fcbb83b-f0fc-427b-806c-62cf4fa66437-credential-socket\") pod \"e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4\" (UID: \"2fcbb83b-f0fc-427b-806c-62cf4fa66437\") " pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4"
Nov 03 18:36:13.059209 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:13.059172    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fcbb83b-f0fc-427b-806c-62cf4fa66437-istio-data\") pod \"e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4\" (UID: \"2fcbb83b-f0fc-427b-806c-62cf4fa66437\") " pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4"
Nov 03 18:36:13.160337 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:13.160240    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fcbb83b-f0fc-427b-806c-62cf4fa66437-istio-podinfo\") pod \"e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4\" (UID: \"2fcbb83b-f0fc-427b-806c-62cf4fa66437\") " pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4"
Nov 03 18:36:13.160337 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:13.160294    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fcbb83b-f0fc-427b-806c-62cf4fa66437-workload-certs\") pod \"e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4\" (UID: \"2fcbb83b-f0fc-427b-806c-62cf4fa66437\") " pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4"
Nov 03 18:36:13.160337 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:13.160329    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fcbb83b-f0fc-427b-806c-62cf4fa66437-istio-token\") pod \"e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4\" (UID: \"2fcbb83b-f0fc-427b-806c-62cf4fa66437\") " pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4"
Nov 03 18:36:13.160702 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:13.160361    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fcbb83b-f0fc-427b-806c-62cf4fa66437-istio-envoy\") pod \"e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4\" (UID: \"2fcbb83b-f0fc-427b-806c-62cf4fa66437\") " pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4"
Nov 03 18:36:13.160702 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:13.160483    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fcbb83b-f0fc-427b-806c-62cf4fa66437-istiod-ca-cert\") pod \"e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4\" (UID: \"2fcbb83b-f0fc-427b-806c-62cf4fa66437\") " pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4"
Nov 03 18:36:13.160702 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:13.160543    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fcbb83b-f0fc-427b-806c-62cf4fa66437-credential-socket\") pod \"e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4\" (UID: \"2fcbb83b-f0fc-427b-806c-62cf4fa66437\") " pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4"
Nov 03 18:36:13.160702 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:13.160581    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fcbb83b-f0fc-427b-806c-62cf4fa66437-istio-data\") pod \"e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4\" (UID: \"2fcbb83b-f0fc-427b-806c-62cf4fa66437\") " pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4"
Nov 03 18:36:13.160702 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:13.160670    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5czh\" (UniqueName: \"kubernetes.io/projected/2fcbb83b-f0fc-427b-806c-62cf4fa66437-kube-api-access-r5czh\") pod \"e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4\" (UID: \"2fcbb83b-f0fc-427b-806c-62cf4fa66437\") " pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4"
Nov 03 18:36:13.160702 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:13.160698    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fcbb83b-f0fc-427b-806c-62cf4fa66437-workload-socket\") pod \"e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4\" (UID: \"2fcbb83b-f0fc-427b-806c-62cf4fa66437\") " pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4"
Nov 03 18:36:13.161062 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:13.160869    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fcbb83b-f0fc-427b-806c-62cf4fa66437-workload-certs\") pod \"e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4\" (UID: \"2fcbb83b-f0fc-427b-806c-62cf4fa66437\") " pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4"
Nov 03 18:36:13.161062 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:13.161030    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fcbb83b-f0fc-427b-806c-62cf4fa66437-workload-socket\") pod \"e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4\" (UID: \"2fcbb83b-f0fc-427b-806c-62cf4fa66437\") " pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4"
Nov 03 18:36:13.161182 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:13.161084    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fcbb83b-f0fc-427b-806c-62cf4fa66437-istio-data\") pod \"e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4\" (UID: \"2fcbb83b-f0fc-427b-806c-62cf4fa66437\") " pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4"
Nov 03 18:36:13.161182 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:13.161099    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fcbb83b-f0fc-427b-806c-62cf4fa66437-credential-socket\") pod \"e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4\" (UID: \"2fcbb83b-f0fc-427b-806c-62cf4fa66437\") " pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4"
Nov 03 18:36:13.161304 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:13.161211    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fcbb83b-f0fc-427b-806c-62cf4fa66437-istiod-ca-cert\") pod \"e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4\" (UID: \"2fcbb83b-f0fc-427b-806c-62cf4fa66437\") " pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4"
Nov 03 18:36:13.162792 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:13.162770    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fcbb83b-f0fc-427b-806c-62cf4fa66437-istio-envoy\") pod \"e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4\" (UID: \"2fcbb83b-f0fc-427b-806c-62cf4fa66437\") " pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4"
Nov 03 18:36:13.162921 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:13.162878    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fcbb83b-f0fc-427b-806c-62cf4fa66437-istio-podinfo\") pod \"e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4\" (UID: \"2fcbb83b-f0fc-427b-806c-62cf4fa66437\") " pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4"
Nov 03 18:36:13.217546 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:13.217509    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fcbb83b-f0fc-427b-806c-62cf4fa66437-istio-token\") pod \"e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4\" (UID: \"2fcbb83b-f0fc-427b-806c-62cf4fa66437\") " pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4"
Nov 03 18:36:13.217802 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:13.217780    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5czh\" (UniqueName: \"kubernetes.io/projected/2fcbb83b-f0fc-427b-806c-62cf4fa66437-kube-api-access-r5czh\") pod \"e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4\" (UID: \"2fcbb83b-f0fc-427b-806c-62cf4fa66437\") " pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4"
Nov 03 18:36:13.490231 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:13.490130    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4"
Nov 03 18:36:13.912460 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:13.912427    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4"]
Nov 03 18:36:13.919683 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:36:13.919650    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fcbb83b_f0fc_427b_806c_62cf4fa66437.slice/crio-02daaef0870e292cee8a0db171b3520be94bef5a2d8b9c0610e6a831728d968b WatchSource:0}: Error finding container 02daaef0870e292cee8a0db171b3520be94bef5a2d8b9c0610e6a831728d968b: Status 404 returned error can't find the container with id 02daaef0870e292cee8a0db171b3520be94bef5a2d8b9c0610e6a831728d968b
Nov 03 18:36:13.922435 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:13.922398    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:36:13.922525 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:13.922470    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:36:13.922525 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:13.922502    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:36:13.928159 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:13.928136    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4" event={"ID":"2fcbb83b-f0fc-427b-806c-62cf4fa66437","Type":"ContainerStarted","Data":"02daaef0870e292cee8a0db171b3520be94bef5a2d8b9c0610e6a831728d968b"}
Nov 03 18:36:14.934951 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:14.934915    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4" event={"ID":"2fcbb83b-f0fc-427b-806c-62cf4fa66437","Type":"ContainerStarted","Data":"13da256411e9047b7367cd5e659d2b2ef7824b0e3fc97ea4e058d942cf99e18c"}
Nov 03 18:36:15.093817 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:15.093767    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4" podStartSLOduration=3.0937524180000002 podStartE2EDuration="3.093752418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:36:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:36:15.092418593 +0000 UTC m=+3014.108616662" watchObservedRunningTime="2026-11-03 18:36:15.093752418 +0000 UTC m=+3014.109950546"
Nov 03 18:36:15.490436 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:15.490393    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4"
Nov 03 18:36:15.495469 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:15.495443    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4"
Nov 03 18:36:15.940681 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:15.940650    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4"
Nov 03 18:36:15.941711 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:15.941691    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4"
Nov 03 18:36:21.041201 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:21.041170    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-07d66926-5fc4d974dc-l8s7l"]
Nov 03 18:36:21.044905 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:21.044878    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-07d66926-5fc4d974dc-l8s7l"
Nov 03 18:36:21.093779 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:21.093738    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-07d66926\""
Nov 03 18:36:21.142310 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:21.142273    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f669d18b-ed53-4d9a-819d-b855956773c3-plugins-config-volume\") pod \"payload-processing-e2e-ait-07d66926-5fc4d974dc-l8s7l\" (UID: \"f669d18b-ed53-4d9a-819d-b855956773c3\") " pod="openshift-ingress/payload-processing-e2e-ait-07d66926-5fc4d974dc-l8s7l"
Nov 03 18:36:21.142517 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:21.142346    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kf97\" (UniqueName: \"kubernetes.io/projected/f669d18b-ed53-4d9a-819d-b855956773c3-kube-api-access-9kf97\") pod \"payload-processing-e2e-ait-07d66926-5fc4d974dc-l8s7l\" (UID: \"f669d18b-ed53-4d9a-819d-b855956773c3\") " pod="openshift-ingress/payload-processing-e2e-ait-07d66926-5fc4d974dc-l8s7l"
Nov 03 18:36:21.144347 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:21.144315    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-07d66926-5fc4d974dc-l8s7l"]
Nov 03 18:36:21.153895 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:21.153862    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-07d66926-dockercfg-nbhxk\""
Nov 03 18:36:21.226053 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:21.226016    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-07d66926-68f54cb575-s9lhb"]
Nov 03 18:36:21.229584 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:21.229558    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-07d66926-68f54cb575-s9lhb"
Nov 03 18:36:21.243261 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:21.243234    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f669d18b-ed53-4d9a-819d-b855956773c3-plugins-config-volume\") pod \"payload-processing-e2e-ait-07d66926-5fc4d974dc-l8s7l\" (UID: \"f669d18b-ed53-4d9a-819d-b855956773c3\") " pod="openshift-ingress/payload-processing-e2e-ait-07d66926-5fc4d974dc-l8s7l"
Nov 03 18:36:21.243409 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:21.243289    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kf97\" (UniqueName: \"kubernetes.io/projected/f669d18b-ed53-4d9a-819d-b855956773c3-kube-api-access-9kf97\") pod \"payload-processing-e2e-ait-07d66926-5fc4d974dc-l8s7l\" (UID: \"f669d18b-ed53-4d9a-819d-b855956773c3\") " pod="openshift-ingress/payload-processing-e2e-ait-07d66926-5fc4d974dc-l8s7l"
Nov 03 18:36:21.243907 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:21.243886    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f669d18b-ed53-4d9a-819d-b855956773c3-plugins-config-volume\") pod \"payload-processing-e2e-ait-07d66926-5fc4d974dc-l8s7l\" (UID: \"f669d18b-ed53-4d9a-819d-b855956773c3\") " pod="openshift-ingress/payload-processing-e2e-ait-07d66926-5fc4d974dc-l8s7l"
Nov 03 18:36:21.290298 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:21.290264    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-07d66926-68f54cb575-s9lhb"]
Nov 03 18:36:21.302381 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:21.302301    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kf97\" (UniqueName: \"kubernetes.io/projected/f669d18b-ed53-4d9a-819d-b855956773c3-kube-api-access-9kf97\") pod \"payload-processing-e2e-ait-07d66926-5fc4d974dc-l8s7l\" (UID: \"f669d18b-ed53-4d9a-819d-b855956773c3\") " pod="openshift-ingress/payload-processing-e2e-ait-07d66926-5fc4d974dc-l8s7l"
Nov 03 18:36:21.344256 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:21.344221    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb7c3dc7-a4ca-404c-bbb5-64b439817ece-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-07d66926-68f54cb575-s9lhb\" (UID: \"fb7c3dc7-a4ca-404c-bbb5-64b439817ece\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-07d66926-68f54cb575-s9lhb"
Nov 03 18:36:21.344256 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:21.344258    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz6rf\" (UniqueName: \"kubernetes.io/projected/fb7c3dc7-a4ca-404c-bbb5-64b439817ece-kube-api-access-nz6rf\") pod \"payload-pre-processing-e2e-ait-07d66926-68f54cb575-s9lhb\" (UID: \"fb7c3dc7-a4ca-404c-bbb5-64b439817ece\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-07d66926-68f54cb575-s9lhb"
Nov 03 18:36:21.359151 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:21.359115    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-07d66926-5fc4d974dc-l8s7l"
Nov 03 18:36:21.445235 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:21.445196    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb7c3dc7-a4ca-404c-bbb5-64b439817ece-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-07d66926-68f54cb575-s9lhb\" (UID: \"fb7c3dc7-a4ca-404c-bbb5-64b439817ece\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-07d66926-68f54cb575-s9lhb"
Nov 03 18:36:21.445427 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:21.445240    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz6rf\" (UniqueName: \"kubernetes.io/projected/fb7c3dc7-a4ca-404c-bbb5-64b439817ece-kube-api-access-nz6rf\") pod \"payload-pre-processing-e2e-ait-07d66926-68f54cb575-s9lhb\" (UID: \"fb7c3dc7-a4ca-404c-bbb5-64b439817ece\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-07d66926-68f54cb575-s9lhb"
Nov 03 18:36:21.446041 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:21.446018    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb7c3dc7-a4ca-404c-bbb5-64b439817ece-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-07d66926-68f54cb575-s9lhb\" (UID: \"fb7c3dc7-a4ca-404c-bbb5-64b439817ece\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-07d66926-68f54cb575-s9lhb"
Nov 03 18:36:21.502823 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:21.502791    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz6rf\" (UniqueName: \"kubernetes.io/projected/fb7c3dc7-a4ca-404c-bbb5-64b439817ece-kube-api-access-nz6rf\") pod \"payload-pre-processing-e2e-ait-07d66926-68f54cb575-s9lhb\" (UID: \"fb7c3dc7-a4ca-404c-bbb5-64b439817ece\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-07d66926-68f54cb575-s9lhb"
Nov 03 18:36:21.542490 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:21.542452    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-07d66926-68f54cb575-s9lhb"
Nov 03 18:36:21.727368 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:21.727333    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-07d66926-5fc4d974dc-l8s7l"]
Nov 03 18:36:21.734762 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:36:21.734731    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf669d18b_ed53_4d9a_819d_b855956773c3.slice/crio-e61114ef8a8af3aa55f36ccd2556b1c50c173e330372ab420c324d0daf43f6df WatchSource:0}: Error finding container e61114ef8a8af3aa55f36ccd2556b1c50c173e330372ab420c324d0daf43f6df: Status 404 returned error can't find the container with id e61114ef8a8af3aa55f36ccd2556b1c50c173e330372ab420c324d0daf43f6df
Nov 03 18:36:21.913883 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:21.913796    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-07d66926-68f54cb575-s9lhb"]
Nov 03 18:36:21.920885 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:36:21.920845    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb7c3dc7_a4ca_404c_bbb5_64b439817ece.slice/crio-59ed20d06a8b61746541e1fa795738ca8c3b7174e8a2b18a1d1181ab08cd8cc5 WatchSource:0}: Error finding container 59ed20d06a8b61746541e1fa795738ca8c3b7174e8a2b18a1d1181ab08cd8cc5: Status 404 returned error can't find the container with id 59ed20d06a8b61746541e1fa795738ca8c3b7174e8a2b18a1d1181ab08cd8cc5
Nov 03 18:36:21.968794 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:21.968759    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-07d66926-68f54cb575-s9lhb" event={"ID":"fb7c3dc7-a4ca-404c-bbb5-64b439817ece","Type":"ContainerStarted","Data":"59ed20d06a8b61746541e1fa795738ca8c3b7174e8a2b18a1d1181ab08cd8cc5"}
Nov 03 18:36:21.970643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:21.970583    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-07d66926-5fc4d974dc-l8s7l" event={"ID":"f669d18b-ed53-4d9a-819d-b855956773c3","Type":"ContainerStarted","Data":"9699068cbbbb9229eff22d2664147441c1bfd671c04ffb89a35b14c608cb1c23"}
Nov 03 18:36:21.970748 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:21.970660    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-07d66926-5fc4d974dc-l8s7l" event={"ID":"f669d18b-ed53-4d9a-819d-b855956773c3","Type":"ContainerStarted","Data":"e61114ef8a8af3aa55f36ccd2556b1c50c173e330372ab420c324d0daf43f6df"}
Nov 03 18:36:21.970798 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:21.970771    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-07d66926-5fc4d974dc-l8s7l"
Nov 03 18:36:22.079106 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:22.079043    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-07d66926-5fc4d974dc-l8s7l" podStartSLOduration=1.079030779 podStartE2EDuration="1.079030779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:36:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:36:22.078061489 +0000 UTC m=+3021.094259555" watchObservedRunningTime="2026-11-03 18:36:22.079030779 +0000 UTC m=+3021.095228845"
Nov 03 18:36:22.977322 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:22.977283    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-07d66926-68f54cb575-s9lhb" event={"ID":"fb7c3dc7-a4ca-404c-bbb5-64b439817ece","Type":"ContainerStarted","Data":"8f60c1b8761a1e21c4daeff3765d1a1054c1d7c646f80266749655dd9b72073c"}
Nov 03 18:36:22.977533 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:22.977352    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-07d66926-68f54cb575-s9lhb"
Nov 03 18:36:23.139038 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:23.138990    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-07d66926-68f54cb575-s9lhb" podStartSLOduration=2.1389764749999998 podStartE2EDuration="2.138976475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:36:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:36:23.138286834 +0000 UTC m=+3022.154484903" watchObservedRunningTime="2026-11-03 18:36:23.138976475 +0000 UTC m=+3022.155174543"
Nov 03 18:36:24.577433 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:36:24.577395    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:36:32.050911 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:32.050874    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6ndfn"]
Nov 03 18:36:32.096580 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:32.096543    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6ndfn"]
Nov 03 18:36:32.980556 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:32.980524    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-07d66926-5fc4d974dc-l8s7l"
Nov 03 18:36:33.102770 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:33.102726    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36719fa9-a3fc-4308-91b8-b41cd779725a" path="/var/lib/kubelet/pods/36719fa9-a3fc-4308-91b8-b41cd779725a/volumes"
Nov 03 18:36:33.985042 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:33.985013    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-07d66926-68f54cb575-s9lhb"
Nov 03 18:36:39.076274 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:39.076239    5315 generic.go:356] "Generic (PLEG): container finished" podID="2fcbb83b-f0fc-427b-806c-62cf4fa66437" containerID="13da256411e9047b7367cd5e659d2b2ef7824b0e3fc97ea4e058d942cf99e18c" exitCode=0
Nov 03 18:36:39.076863 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:39.076319    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4" event={"ID":"2fcbb83b-f0fc-427b-806c-62cf4fa66437","Type":"ContainerDied","Data":"13da256411e9047b7367cd5e659d2b2ef7824b0e3fc97ea4e058d942cf99e18c"}
Nov 03 18:36:39.076863 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:39.076851    5315 scope.go:122] "RemoveContainer" containerID="13da256411e9047b7367cd5e659d2b2ef7824b0e3fc97ea4e058d942cf99e18c"
Nov 03 18:36:39.078044 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:39.078015    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:36:39.078112 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:39.078090    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:36:39.078150 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:39.078123    5315 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 18:36:40.083412 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:40.083376    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4" event={"ID":"2fcbb83b-f0fc-427b-806c-62cf4fa66437","Type":"ContainerStarted","Data":"e7369636fb8f50e2d921084c0f5107c88bab8488b8749b0acc10f37ff813bb68"}
Nov 03 18:36:40.491354 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:40.491258    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4"
Nov 03 18:36:40.491526 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:40.491498    5315 patch_prober.go:29] interesting pod/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body=
Nov 03 18:36:40.491571 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:40.491542    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4" podUID="2fcbb83b-f0fc-427b-806c-62cf4fa66437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused"
Nov 03 18:36:41.490970 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:41.490934    5315 patch_prober.go:29] interesting pod/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body=
Nov 03 18:36:41.491370 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:41.491001    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4" podUID="2fcbb83b-f0fc-427b-806c-62cf4fa66437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused"
Nov 03 18:36:42.491375 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:42.491335    5315 patch_prober.go:29] interesting pod/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body=
Nov 03 18:36:42.491819 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:42.491386    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4" podUID="2fcbb83b-f0fc-427b-806c-62cf4fa66437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused"
Nov 03 18:36:43.491365 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:43.491325    5315 patch_prober.go:29] interesting pod/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body=
Nov 03 18:36:43.491563 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:43.491393    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4" podUID="2fcbb83b-f0fc-427b-806c-62cf4fa66437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused"
Nov 03 18:36:44.491373 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:44.491325    5315 patch_prober.go:29] interesting pod/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body=
Nov 03 18:36:44.491578 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:44.491403    5315 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4" podUID="2fcbb83b-f0fc-427b-806c-62cf4fa66437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused"
Nov 03 18:36:45.495000 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:45.494966    5315 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4"
Nov 03 18:36:45.495423 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:45.495264    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4"
Nov 03 18:36:45.496002 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:45.495984    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4"
Nov 03 18:36:48.814591 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:48.814553    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-07d66926-l64f7"]
Nov 03 18:36:48.816991 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:48.816972    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-07d66926-l64f7"
Nov 03 18:36:48.867940 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:48.867913    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:36:48.911673 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:48.911640    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:36:48.911673 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:48.911640    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:36:48.911931 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:48.911699    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgm5\""
Nov 03 18:36:48.913303 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:48.913280    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-07d66926-l64f7"]
Nov 03 18:36:48.938917 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:48.938883    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64mnv\" (UniqueName: \"kubernetes.io/projected/5f241df0-9515-4151-8da5-de0d2a768335-kube-api-access-64mnv\") pod \"maas-api-revoke-keys-e2e-ait-07d66926-l64f7\" (UID: \"5f241df0-9515-4151-8da5-de0d2a768335\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-07d66926-l64f7"
Nov 03 18:36:48.939094 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:48.938938    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f241df0-9515-4151-8da5-de0d2a768335-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-07d66926-l64f7\" (UID: \"5f241df0-9515-4151-8da5-de0d2a768335\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-07d66926-l64f7"
Nov 03 18:36:48.939094 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:48.939005    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f241df0-9515-4151-8da5-de0d2a768335-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-07d66926-l64f7\" (UID: \"5f241df0-9515-4151-8da5-de0d2a768335\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-07d66926-l64f7"
Nov 03 18:36:49.039960 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:49.039919    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64mnv\" (UniqueName: \"kubernetes.io/projected/5f241df0-9515-4151-8da5-de0d2a768335-kube-api-access-64mnv\") pod \"maas-api-revoke-keys-e2e-ait-07d66926-l64f7\" (UID: \"5f241df0-9515-4151-8da5-de0d2a768335\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-07d66926-l64f7"
Nov 03 18:36:49.040151 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:49.039980    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f241df0-9515-4151-8da5-de0d2a768335-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-07d66926-l64f7\" (UID: \"5f241df0-9515-4151-8da5-de0d2a768335\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-07d66926-l64f7"
Nov 03 18:36:49.040151 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:49.040012    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f241df0-9515-4151-8da5-de0d2a768335-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-07d66926-l64f7\" (UID: \"5f241df0-9515-4151-8da5-de0d2a768335\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-07d66926-l64f7"
Nov 03 18:36:49.040828 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:49.040806    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f241df0-9515-4151-8da5-de0d2a768335-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-07d66926-l64f7\" (UID: \"5f241df0-9515-4151-8da5-de0d2a768335\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-07d66926-l64f7"
Nov 03 18:36:49.042394 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:49.042374    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f241df0-9515-4151-8da5-de0d2a768335-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-07d66926-l64f7\" (UID: \"5f241df0-9515-4151-8da5-de0d2a768335\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-07d66926-l64f7"
Nov 03 18:36:49.096845 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:49.096768    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64mnv\" (UniqueName: \"kubernetes.io/projected/5f241df0-9515-4151-8da5-de0d2a768335-kube-api-access-64mnv\") pod \"maas-api-revoke-keys-e2e-ait-07d66926-l64f7\" (UID: \"5f241df0-9515-4151-8da5-de0d2a768335\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-07d66926-l64f7"
Nov 03 18:36:49.129106 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:49.129073    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-07d66926-l64f7"
Nov 03 18:36:49.493739 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:49.493651    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-07d66926-l64f7"]
Nov 03 18:36:49.500674 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:36:49.500643    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f241df0_9515_4151_8da5_de0d2a768335.slice/crio-66d700ac4bd4546a91cc7c0ee27e303a4c12a530accaa42686221b253efa95b4 WatchSource:0}: Error finding container 66d700ac4bd4546a91cc7c0ee27e303a4c12a530accaa42686221b253efa95b4: Status 404 returned error can't find the container with id 66d700ac4bd4546a91cc7c0ee27e303a4c12a530accaa42686221b253efa95b4
Nov 03 18:36:50.130534 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:50.130440    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-07d66926-l64f7" event={"ID":"5f241df0-9515-4151-8da5-de0d2a768335","Type":"ContainerStarted","Data":"528aacacd9cea3542e3db5e4809c2951234830dd25ccf21d3392b66020516674"}
Nov 03 18:36:50.130534 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:50.130507    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-07d66926-l64f7" event={"ID":"5f241df0-9515-4151-8da5-de0d2a768335","Type":"ContainerStarted","Data":"66d700ac4bd4546a91cc7c0ee27e303a4c12a530accaa42686221b253efa95b4"}
Nov 03 18:36:50.238845 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:50.238793    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-07d66926-l64f7" podStartSLOduration=2.238780702 podStartE2EDuration="2.238780702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:36:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:36:50.238382437 +0000 UTC m=+3049.254580507" watchObservedRunningTime="2026-11-03 18:36:50.238780702 +0000 UTC m=+3049.254978770"
Nov 03 18:36:51.136740 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:51.136706    5315 generic.go:356] "Generic (PLEG): container finished" podID="5f241df0-9515-4151-8da5-de0d2a768335" containerID="528aacacd9cea3542e3db5e4809c2951234830dd25ccf21d3392b66020516674" exitCode=0
Nov 03 18:36:51.137172 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:51.136799    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-07d66926-l64f7" event={"ID":"5f241df0-9515-4151-8da5-de0d2a768335","Type":"ContainerDied","Data":"528aacacd9cea3542e3db5e4809c2951234830dd25ccf21d3392b66020516674"}
Nov 03 18:36:52.272385 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:52.272351    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-07d66926-l64f7"
Nov 03 18:36:52.373750 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:52.373714    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f241df0-9515-4151-8da5-de0d2a768335-kube-api-access-64mnv\" (UniqueName: \"kubernetes.io/projected/5f241df0-9515-4151-8da5-de0d2a768335-kube-api-access-64mnv\") pod \"5f241df0-9515-4151-8da5-de0d2a768335\" (UID: \"5f241df0-9515-4151-8da5-de0d2a768335\") "
Nov 03 18:36:52.373923 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:52.373860    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f241df0-9515-4151-8da5-de0d2a768335-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f241df0-9515-4151-8da5-de0d2a768335-maas-api-xks-ca\") pod \"5f241df0-9515-4151-8da5-de0d2a768335\" (UID: \"5f241df0-9515-4151-8da5-de0d2a768335\") "
Nov 03 18:36:52.373923 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:52.373897    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f241df0-9515-4151-8da5-de0d2a768335-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f241df0-9515-4151-8da5-de0d2a768335-maas-api-openshift-ca\") pod \"5f241df0-9515-4151-8da5-de0d2a768335\" (UID: \"5f241df0-9515-4151-8da5-de0d2a768335\") "
Nov 03 18:36:52.374330 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:52.374303    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f241df0-9515-4151-8da5-de0d2a768335-maas-api-openshift-ca" pod "5f241df0-9515-4151-8da5-de0d2a768335" (UID: "5f241df0-9515-4151-8da5-de0d2a768335"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:36:52.375847 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:52.375816    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f241df0-9515-4151-8da5-de0d2a768335-kube-api-access-64mnv" pod "5f241df0-9515-4151-8da5-de0d2a768335" (UID: "5f241df0-9515-4151-8da5-de0d2a768335"). InnerVolumeSpecName "kube-api-access-64mnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:36:52.375958 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:52.375914    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f241df0-9515-4151-8da5-de0d2a768335-maas-api-xks-ca" pod "5f241df0-9515-4151-8da5-de0d2a768335" (UID: "5f241df0-9515-4151-8da5-de0d2a768335"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:36:52.474988 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:52.474897    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f241df0-9515-4151-8da5-de0d2a768335-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:36:52.474988 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:52.474933    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f241df0-9515-4151-8da5-de0d2a768335-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:36:52.474988 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:52.474949    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64mnv\" (UniqueName: \"kubernetes.io/projected/5f241df0-9515-4151-8da5-de0d2a768335-kube-api-access-64mnv\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:36:53.147768 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:53.147730    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-07d66926-l64f7" event={"ID":"5f241df0-9515-4151-8da5-de0d2a768335","Type":"ContainerDied","Data":"66d700ac4bd4546a91cc7c0ee27e303a4c12a530accaa42686221b253efa95b4"}
Nov 03 18:36:53.147768 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:53.147764    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-07d66926-l64f7"
Nov 03 18:36:53.148048 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:36:53.147785    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66d700ac4bd4546a91cc7c0ee27e303a4c12a530accaa42686221b253efa95b4"
Nov 03 18:36:54.577055 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:36:54.577019    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:37:00.246455 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:00.246414    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-07d66926-5fc4d974dc-l8s7l"]
Nov 03 18:37:00.342172 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:00.342122    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-07d66926-68f54cb575-s9lhb"]
Nov 03 18:37:04.903943 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:04.903896    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-07d66926-l64f7"]
Nov 03 18:37:04.949825 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:04.949789    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-07d66926-l64f7"]
Nov 03 18:37:05.102648 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:05.102594    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f241df0-9515-4151-8da5-de0d2a768335" path="/var/lib/kubelet/pods/5f241df0-9515-4151-8da5-de0d2a768335/volumes"
Nov 03 18:37:05.246814 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:05.246708    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-07d66926-5fc4d974dc-l8s7l" podUID="f669d18b-ed53-4d9a-819d-b855956773c3" containerName="payload-processing" containerID="cri-o://9699068cbbbb9229eff22d2664147441c1bfd671c04ffb89a35b14c608cb1c23" gracePeriod=25
Nov 03 18:37:05.342808 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:05.342752    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-07d66926-68f54cb575-s9lhb" podUID="fb7c3dc7-a4ca-404c-bbb5-64b439817ece" containerName="payload-pre-processing" containerID="cri-o://8f60c1b8761a1e21c4daeff3765d1a1054c1d7c646f80266749655dd9b72073c" gracePeriod=25
Nov 03 18:37:05.585115 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:05.585085    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-07d66926-5fc4d974dc-l8s7l"
Nov 03 18:37:05.614253 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:05.614223    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-07d66926-68f54cb575-s9lhb"
Nov 03 18:37:05.718245 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:05.718210    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f669d18b-ed53-4d9a-819d-b855956773c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f669d18b-ed53-4d9a-819d-b855956773c3-plugins-config-volume\") pod \"f669d18b-ed53-4d9a-819d-b855956773c3\" (UID: \"f669d18b-ed53-4d9a-819d-b855956773c3\") "
Nov 03 18:37:05.718441 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:05.718290    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb7c3dc7-a4ca-404c-bbb5-64b439817ece-kube-api-access-nz6rf\" (UniqueName: \"kubernetes.io/projected/fb7c3dc7-a4ca-404c-bbb5-64b439817ece-kube-api-access-nz6rf\") pod \"fb7c3dc7-a4ca-404c-bbb5-64b439817ece\" (UID: \"fb7c3dc7-a4ca-404c-bbb5-64b439817ece\") "
Nov 03 18:37:05.718441 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:05.718363    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f669d18b-ed53-4d9a-819d-b855956773c3-kube-api-access-9kf97\" (UniqueName: \"kubernetes.io/projected/f669d18b-ed53-4d9a-819d-b855956773c3-kube-api-access-9kf97\") pod \"f669d18b-ed53-4d9a-819d-b855956773c3\" (UID: \"f669d18b-ed53-4d9a-819d-b855956773c3\") "
Nov 03 18:37:05.718441 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:05.718389    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb7c3dc7-a4ca-404c-bbb5-64b439817ece-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb7c3dc7-a4ca-404c-bbb5-64b439817ece-plugins-config-volume\") pod \"fb7c3dc7-a4ca-404c-bbb5-64b439817ece\" (UID: \"fb7c3dc7-a4ca-404c-bbb5-64b439817ece\") "
Nov 03 18:37:05.718737 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:05.718689    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f669d18b-ed53-4d9a-819d-b855956773c3-plugins-config-volume" pod "f669d18b-ed53-4d9a-819d-b855956773c3" (UID: "f669d18b-ed53-4d9a-819d-b855956773c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:37:05.718889 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:05.718862    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb7c3dc7-a4ca-404c-bbb5-64b439817ece-plugins-config-volume" pod "fb7c3dc7-a4ca-404c-bbb5-64b439817ece" (UID: "fb7c3dc7-a4ca-404c-bbb5-64b439817ece"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:37:05.720551 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:05.720523    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f669d18b-ed53-4d9a-819d-b855956773c3-kube-api-access-9kf97" pod "f669d18b-ed53-4d9a-819d-b855956773c3" (UID: "f669d18b-ed53-4d9a-819d-b855956773c3"). InnerVolumeSpecName "kube-api-access-9kf97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:37:05.720675 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:05.720570    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb7c3dc7-a4ca-404c-bbb5-64b439817ece-kube-api-access-nz6rf" pod "fb7c3dc7-a4ca-404c-bbb5-64b439817ece" (UID: "fb7c3dc7-a4ca-404c-bbb5-64b439817ece"). InnerVolumeSpecName "kube-api-access-nz6rf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:37:05.819574 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:05.819534    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz6rf\" (UniqueName: \"kubernetes.io/projected/fb7c3dc7-a4ca-404c-bbb5-64b439817ece-kube-api-access-nz6rf\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:37:05.819574 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:05.819573    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kf97\" (UniqueName: \"kubernetes.io/projected/f669d18b-ed53-4d9a-819d-b855956773c3-kube-api-access-9kf97\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:37:05.819870 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:05.819590    5315 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb7c3dc7-a4ca-404c-bbb5-64b439817ece-plugins-config-volume\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:37:05.819870 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:05.819621    5315 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f669d18b-ed53-4d9a-819d-b855956773c3-plugins-config-volume\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:37:06.229057 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:06.228975    5315 generic.go:356] "Generic (PLEG): container finished" podID="fb7c3dc7-a4ca-404c-bbb5-64b439817ece" containerID="8f60c1b8761a1e21c4daeff3765d1a1054c1d7c646f80266749655dd9b72073c" exitCode=0
Nov 03 18:37:06.229057 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:06.229016    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-07d66926-68f54cb575-s9lhb"
Nov 03 18:37:06.229574 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:06.229067    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-07d66926-68f54cb575-s9lhb" event={"ID":"fb7c3dc7-a4ca-404c-bbb5-64b439817ece","Type":"ContainerDied","Data":"8f60c1b8761a1e21c4daeff3765d1a1054c1d7c646f80266749655dd9b72073c"}
Nov 03 18:37:06.229574 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:06.229118    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-07d66926-68f54cb575-s9lhb" event={"ID":"fb7c3dc7-a4ca-404c-bbb5-64b439817ece","Type":"ContainerDied","Data":"59ed20d06a8b61746541e1fa795738ca8c3b7174e8a2b18a1d1181ab08cd8cc5"}
Nov 03 18:37:06.229574 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:06.229146    5315 scope.go:122] "RemoveContainer" containerID="8f60c1b8761a1e21c4daeff3765d1a1054c1d7c646f80266749655dd9b72073c"
Nov 03 18:37:06.230490 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:06.230469    5315 generic.go:356] "Generic (PLEG): container finished" podID="f669d18b-ed53-4d9a-819d-b855956773c3" containerID="9699068cbbbb9229eff22d2664147441c1bfd671c04ffb89a35b14c608cb1c23" exitCode=0
Nov 03 18:37:06.230577 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:06.230521    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-07d66926-5fc4d974dc-l8s7l"
Nov 03 18:37:06.230650 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:06.230522    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-07d66926-5fc4d974dc-l8s7l" event={"ID":"f669d18b-ed53-4d9a-819d-b855956773c3","Type":"ContainerDied","Data":"9699068cbbbb9229eff22d2664147441c1bfd671c04ffb89a35b14c608cb1c23"}
Nov 03 18:37:06.230699 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:06.230656    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-07d66926-5fc4d974dc-l8s7l" event={"ID":"f669d18b-ed53-4d9a-819d-b855956773c3","Type":"ContainerDied","Data":"e61114ef8a8af3aa55f36ccd2556b1c50c173e330372ab420c324d0daf43f6df"}
Nov 03 18:37:06.237856 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:06.237837    5315 scope.go:122] "RemoveContainer" containerID="8f60c1b8761a1e21c4daeff3765d1a1054c1d7c646f80266749655dd9b72073c"
Nov 03 18:37:06.238137 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:37:06.238117    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f60c1b8761a1e21c4daeff3765d1a1054c1d7c646f80266749655dd9b72073c\": container with ID starting with 8f60c1b8761a1e21c4daeff3765d1a1054c1d7c646f80266749655dd9b72073c not found: ID does not exist" containerID="8f60c1b8761a1e21c4daeff3765d1a1054c1d7c646f80266749655dd9b72073c"
Nov 03 18:37:06.238210 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:06.238145    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f60c1b8761a1e21c4daeff3765d1a1054c1d7c646f80266749655dd9b72073c"} err="failed to get container status \"8f60c1b8761a1e21c4daeff3765d1a1054c1d7c646f80266749655dd9b72073c\": rpc error: code = NotFound desc = could not find container \"8f60c1b8761a1e21c4daeff3765d1a1054c1d7c646f80266749655dd9b72073c\": container with ID starting with 8f60c1b8761a1e21c4daeff3765d1a1054c1d7c646f80266749655dd9b72073c not found: ID does not exist"
Nov 03 18:37:06.238210 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:06.238161    5315 scope.go:122] "RemoveContainer" containerID="9699068cbbbb9229eff22d2664147441c1bfd671c04ffb89a35b14c608cb1c23"
Nov 03 18:37:06.246596 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:06.246576    5315 scope.go:122] "RemoveContainer" containerID="9699068cbbbb9229eff22d2664147441c1bfd671c04ffb89a35b14c608cb1c23"
Nov 03 18:37:06.246886 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:37:06.246866    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9699068cbbbb9229eff22d2664147441c1bfd671c04ffb89a35b14c608cb1c23\": container with ID starting with 9699068cbbbb9229eff22d2664147441c1bfd671c04ffb89a35b14c608cb1c23 not found: ID does not exist" containerID="9699068cbbbb9229eff22d2664147441c1bfd671c04ffb89a35b14c608cb1c23"
Nov 03 18:37:06.246938 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:06.246895    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9699068cbbbb9229eff22d2664147441c1bfd671c04ffb89a35b14c608cb1c23"} err="failed to get container status \"9699068cbbbb9229eff22d2664147441c1bfd671c04ffb89a35b14c608cb1c23\": rpc error: code = NotFound desc = could not find container \"9699068cbbbb9229eff22d2664147441c1bfd671c04ffb89a35b14c608cb1c23\": container with ID starting with 9699068cbbbb9229eff22d2664147441c1bfd671c04ffb89a35b14c608cb1c23 not found: ID does not exist"
Nov 03 18:37:06.380373 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:06.380330    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-07d66926-5fc4d974dc-l8s7l"]
Nov 03 18:37:06.384750 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:06.384723    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-07d66926-5fc4d974dc-l8s7l"]
Nov 03 18:37:06.461632 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:06.461585    5315 scope.go:122] "RemoveContainer" containerID="349ff2e01710edb58b42adfe29eaa95ee041a78a6319a3f2f914caeaed1823cd"
Nov 03 18:37:06.486283 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:06.486260    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-07d66926-68f54cb575-s9lhb"]
Nov 03 18:37:06.490467 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:06.490444    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-07d66926-68f54cb575-s9lhb"]
Nov 03 18:37:07.102825 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:07.102792    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f669d18b-ed53-4d9a-819d-b855956773c3" path="/var/lib/kubelet/pods/f669d18b-ed53-4d9a-819d-b855956773c3/volumes"
Nov 03 18:37:07.103168 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:07.103155    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb7c3dc7-a4ca-404c-bbb5-64b439817ece" path="/var/lib/kubelet/pods/fb7c3dc7-a4ca-404c-bbb5-64b439817ece/volumes"
Nov 03 18:37:09.800017 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:09.799982    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4"]
Nov 03 18:37:09.800431 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:09.800208    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4" podUID="2fcbb83b-f0fc-427b-806c-62cf4fa66437" containerName="istio-proxy" containerID="cri-o://e7369636fb8f50e2d921084c0f5107c88bab8488b8749b0acc10f37ff813bb68" gracePeriod=30
Nov 03 18:37:15.046994 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:15.046961    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4"
Nov 03 18:37:15.116665 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:15.116565    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fcbb83b-f0fc-427b-806c-62cf4fa66437-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fcbb83b-f0fc-427b-806c-62cf4fa66437-istio-envoy\") pod \"2fcbb83b-f0fc-427b-806c-62cf4fa66437\" (UID: \"2fcbb83b-f0fc-427b-806c-62cf4fa66437\") "
Nov 03 18:37:15.116665 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:15.116601    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fcbb83b-f0fc-427b-806c-62cf4fa66437-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fcbb83b-f0fc-427b-806c-62cf4fa66437-istio-data\") pod \"2fcbb83b-f0fc-427b-806c-62cf4fa66437\" (UID: \"2fcbb83b-f0fc-427b-806c-62cf4fa66437\") "
Nov 03 18:37:15.116665 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:15.116656    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fcbb83b-f0fc-427b-806c-62cf4fa66437-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fcbb83b-f0fc-427b-806c-62cf4fa66437-credential-socket\") pod \"2fcbb83b-f0fc-427b-806c-62cf4fa66437\" (UID: \"2fcbb83b-f0fc-427b-806c-62cf4fa66437\") "
Nov 03 18:37:15.116909 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:15.116745    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fcbb83b-f0fc-427b-806c-62cf4fa66437-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fcbb83b-f0fc-427b-806c-62cf4fa66437-istiod-ca-cert\") pod \"2fcbb83b-f0fc-427b-806c-62cf4fa66437\" (UID: \"2fcbb83b-f0fc-427b-806c-62cf4fa66437\") "
Nov 03 18:37:15.116909 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:15.116786    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fcbb83b-f0fc-427b-806c-62cf4fa66437-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fcbb83b-f0fc-427b-806c-62cf4fa66437-workload-certs\") pod \"2fcbb83b-f0fc-427b-806c-62cf4fa66437\" (UID: \"2fcbb83b-f0fc-427b-806c-62cf4fa66437\") "
Nov 03 18:37:15.116909 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:15.116846    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fcbb83b-f0fc-427b-806c-62cf4fa66437-kube-api-access-r5czh\" (UniqueName: \"kubernetes.io/projected/2fcbb83b-f0fc-427b-806c-62cf4fa66437-kube-api-access-r5czh\") pod \"2fcbb83b-f0fc-427b-806c-62cf4fa66437\" (UID: \"2fcbb83b-f0fc-427b-806c-62cf4fa66437\") "
Nov 03 18:37:15.116909 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:15.116892    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fcbb83b-f0fc-427b-806c-62cf4fa66437-istio-token\" (UniqueName: \"kubernetes.io/projected/2fcbb83b-f0fc-427b-806c-62cf4fa66437-istio-token\") pod \"2fcbb83b-f0fc-427b-806c-62cf4fa66437\" (UID: \"2fcbb83b-f0fc-427b-806c-62cf4fa66437\") "
Nov 03 18:37:15.117114 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:15.116925    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fcbb83b-f0fc-427b-806c-62cf4fa66437-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fcbb83b-f0fc-427b-806c-62cf4fa66437-workload-socket\") pod \"2fcbb83b-f0fc-427b-806c-62cf4fa66437\" (UID: \"2fcbb83b-f0fc-427b-806c-62cf4fa66437\") "
Nov 03 18:37:15.117114 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:15.116971    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2fcbb83b-f0fc-427b-806c-62cf4fa66437-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fcbb83b-f0fc-427b-806c-62cf4fa66437-istio-podinfo\") pod \"2fcbb83b-f0fc-427b-806c-62cf4fa66437\" (UID: \"2fcbb83b-f0fc-427b-806c-62cf4fa66437\") "
Nov 03 18:37:15.117114 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:15.116956    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fcbb83b-f0fc-427b-806c-62cf4fa66437-istio-data" pod "2fcbb83b-f0fc-427b-806c-62cf4fa66437" (UID: "2fcbb83b-f0fc-427b-806c-62cf4fa66437"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:37:15.117289 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:15.117163    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fcbb83b-f0fc-427b-806c-62cf4fa66437-workload-certs" pod "2fcbb83b-f0fc-427b-806c-62cf4fa66437" (UID: "2fcbb83b-f0fc-427b-806c-62cf4fa66437"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:37:15.117289 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:15.117179    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fcbb83b-f0fc-427b-806c-62cf4fa66437-workload-socket" pod "2fcbb83b-f0fc-427b-806c-62cf4fa66437" (UID: "2fcbb83b-f0fc-427b-806c-62cf4fa66437"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:37:15.117289 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:15.117238    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fcbb83b-f0fc-427b-806c-62cf4fa66437-credential-socket" pod "2fcbb83b-f0fc-427b-806c-62cf4fa66437" (UID: "2fcbb83b-f0fc-427b-806c-62cf4fa66437"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:37:15.117444 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:15.117293    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fcbb83b-f0fc-427b-806c-62cf4fa66437-istiod-ca-cert" pod "2fcbb83b-f0fc-427b-806c-62cf4fa66437" (UID: "2fcbb83b-f0fc-427b-806c-62cf4fa66437"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:37:15.117444 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:15.117331    5315 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fcbb83b-f0fc-427b-806c-62cf4fa66437-workload-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:37:15.117444 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:15.117367    5315 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fcbb83b-f0fc-427b-806c-62cf4fa66437-istio-data\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:37:15.117444 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:15.117376    5315 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fcbb83b-f0fc-427b-806c-62cf4fa66437-credential-socket\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:37:15.117444 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:15.117386    5315 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fcbb83b-f0fc-427b-806c-62cf4fa66437-workload-certs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:37:15.118909 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:15.118887    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fcbb83b-f0fc-427b-806c-62cf4fa66437-istio-envoy" pod "2fcbb83b-f0fc-427b-806c-62cf4fa66437" (UID: "2fcbb83b-f0fc-427b-806c-62cf4fa66437"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:37:15.119137 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:15.119121    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2fcbb83b-f0fc-427b-806c-62cf4fa66437-istio-podinfo" pod "2fcbb83b-f0fc-427b-806c-62cf4fa66437" (UID: "2fcbb83b-f0fc-427b-806c-62cf4fa66437"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:37:15.119402 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:15.119374    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fcbb83b-f0fc-427b-806c-62cf4fa66437-istio-token" pod "2fcbb83b-f0fc-427b-806c-62cf4fa66437" (UID: "2fcbb83b-f0fc-427b-806c-62cf4fa66437"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:37:15.119501 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:15.119419    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fcbb83b-f0fc-427b-806c-62cf4fa66437-kube-api-access-r5czh" pod "2fcbb83b-f0fc-427b-806c-62cf4fa66437" (UID: "2fcbb83b-f0fc-427b-806c-62cf4fa66437"). InnerVolumeSpecName "kube-api-access-r5czh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:37:15.218051 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:15.218017    5315 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fcbb83b-f0fc-427b-806c-62cf4fa66437-istiod-ca-cert\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:37:15.218051 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:15.218044    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5czh\" (UniqueName: \"kubernetes.io/projected/2fcbb83b-f0fc-427b-806c-62cf4fa66437-kube-api-access-r5czh\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:37:15.218051 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:15.218054    5315 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fcbb83b-f0fc-427b-806c-62cf4fa66437-istio-token\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:37:15.218310 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:15.218064    5315 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fcbb83b-f0fc-427b-806c-62cf4fa66437-istio-podinfo\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:37:15.218310 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:15.218073    5315 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fcbb83b-f0fc-427b-806c-62cf4fa66437-istio-envoy\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:37:15.273454 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:15.273408    5315 generic.go:356] "Generic (PLEG): container finished" podID="2fcbb83b-f0fc-427b-806c-62cf4fa66437" containerID="e7369636fb8f50e2d921084c0f5107c88bab8488b8749b0acc10f37ff813bb68" exitCode=0
Nov 03 18:37:15.273939 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:15.273459    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4"
Nov 03 18:37:15.273939 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:15.273484    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4" event={"ID":"2fcbb83b-f0fc-427b-806c-62cf4fa66437","Type":"ContainerDied","Data":"e7369636fb8f50e2d921084c0f5107c88bab8488b8749b0acc10f37ff813bb68"}
Nov 03 18:37:15.273939 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:15.273541    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4" event={"ID":"2fcbb83b-f0fc-427b-806c-62cf4fa66437","Type":"ContainerDied","Data":"02daaef0870e292cee8a0db171b3520be94bef5a2d8b9c0610e6a831728d968b"}
Nov 03 18:37:15.273939 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:15.273564    5315 scope.go:122] "RemoveContainer" containerID="e7369636fb8f50e2d921084c0f5107c88bab8488b8749b0acc10f37ff813bb68"
Nov 03 18:37:15.283111 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:15.283091    5315 scope.go:122] "RemoveContainer" containerID="13da256411e9047b7367cd5e659d2b2ef7824b0e3fc97ea4e058d942cf99e18c"
Nov 03 18:37:15.291540 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:15.291518    5315 scope.go:122] "RemoveContainer" containerID="e7369636fb8f50e2d921084c0f5107c88bab8488b8749b0acc10f37ff813bb68"
Nov 03 18:37:15.291837 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:37:15.291814    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7369636fb8f50e2d921084c0f5107c88bab8488b8749b0acc10f37ff813bb68\": container with ID starting with e7369636fb8f50e2d921084c0f5107c88bab8488b8749b0acc10f37ff813bb68 not found: ID does not exist" containerID="e7369636fb8f50e2d921084c0f5107c88bab8488b8749b0acc10f37ff813bb68"
Nov 03 18:37:15.291934 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:15.291845    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7369636fb8f50e2d921084c0f5107c88bab8488b8749b0acc10f37ff813bb68"} err="failed to get container status \"e7369636fb8f50e2d921084c0f5107c88bab8488b8749b0acc10f37ff813bb68\": rpc error: code = NotFound desc = could not find container \"e7369636fb8f50e2d921084c0f5107c88bab8488b8749b0acc10f37ff813bb68\": container with ID starting with e7369636fb8f50e2d921084c0f5107c88bab8488b8749b0acc10f37ff813bb68 not found: ID does not exist"
Nov 03 18:37:15.291934 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:15.291862    5315 scope.go:122] "RemoveContainer" containerID="13da256411e9047b7367cd5e659d2b2ef7824b0e3fc97ea4e058d942cf99e18c"
Nov 03 18:37:15.292090 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:37:15.292075    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13da256411e9047b7367cd5e659d2b2ef7824b0e3fc97ea4e058d942cf99e18c\": container with ID starting with 13da256411e9047b7367cd5e659d2b2ef7824b0e3fc97ea4e058d942cf99e18c not found: ID does not exist" containerID="13da256411e9047b7367cd5e659d2b2ef7824b0e3fc97ea4e058d942cf99e18c"
Nov 03 18:37:15.292136 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:15.292094    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13da256411e9047b7367cd5e659d2b2ef7824b0e3fc97ea4e058d942cf99e18c"} err="failed to get container status \"13da256411e9047b7367cd5e659d2b2ef7824b0e3fc97ea4e058d942cf99e18c\": rpc error: code = NotFound desc = could not find container \"13da256411e9047b7367cd5e659d2b2ef7824b0e3fc97ea4e058d942cf99e18c\": container with ID starting with 13da256411e9047b7367cd5e659d2b2ef7824b0e3fc97ea4e058d942cf99e18c not found: ID does not exist"
Nov 03 18:37:15.427210 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:15.427123    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4"]
Nov 03 18:37:15.431171 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:15.431138    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-07d66926-openshift-default-667cf7dd89-9pgx4"]
Nov 03 18:37:17.103320 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:17.103284    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fcbb83b-f0fc-427b-806c-62cf4fa66437" path="/var/lib/kubelet/pods/2fcbb83b-f0fc-427b-806c-62cf4fa66437/volumes"
Nov 03 18:37:19.973422 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:19.973381    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-2300af-64b4d74fff-q2xp7"]
Nov 03 18:37:19.974151 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:19.974113    5315 cpu_manager.go:414] "containerMap: removing container" podUID="f669d18b-ed53-4d9a-819d-b855956773c3" containerName="payload-processing"
Nov 03 18:37:19.974151 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:19.974139    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f669d18b-ed53-4d9a-819d-b855956773c3" containerName="payload-processing"
Nov 03 18:37:19.974370 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:19.974170    5315 cpu_manager.go:414] "containerMap: removing container" podUID="5f241df0-9515-4151-8da5-de0d2a768335" containerName="revoke-keys"
Nov 03 18:37:19.974370 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:19.974179    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f241df0-9515-4151-8da5-de0d2a768335" containerName="revoke-keys"
Nov 03 18:37:19.974370 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:19.974198    5315 cpu_manager.go:414] "containerMap: removing container" podUID="2fcbb83b-f0fc-427b-806c-62cf4fa66437" containerName="istio-proxy"
Nov 03 18:37:19.974370 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:19.974206    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fcbb83b-f0fc-427b-806c-62cf4fa66437" containerName="istio-proxy"
Nov 03 18:37:19.974370 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:19.974233    5315 cpu_manager.go:414] "containerMap: removing container" podUID="fb7c3dc7-a4ca-404c-bbb5-64b439817ece" containerName="payload-pre-processing"
Nov 03 18:37:19.974370 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:19.974240    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb7c3dc7-a4ca-404c-bbb5-64b439817ece" containerName="payload-pre-processing"
Nov 03 18:37:19.974370 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:19.974334    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fcbb83b-f0fc-427b-806c-62cf4fa66437" containerName="istio-proxy"
Nov 03 18:37:19.974370 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:19.974346    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="f669d18b-ed53-4d9a-819d-b855956773c3" containerName="payload-processing"
Nov 03 18:37:19.974370 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:19.974360    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f241df0-9515-4151-8da5-de0d2a768335" containerName="revoke-keys"
Nov 03 18:37:19.974370 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:19.974375    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb7c3dc7-a4ca-404c-bbb5-64b439817ece" containerName="payload-pre-processing"
Nov 03 18:37:19.987643 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:19.987597    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-2300af-64b4d74fff-q2xp7"
Nov 03 18:37:20.036344 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:20.036307    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-2300af-64b4d74fff-q2xp7"]
Nov 03 18:37:20.059657 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:20.059595    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bcd7af3-c2f8-48a2-ae11-a7cce16f429e-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-2300af-64b4d74fff-q2xp7\" (UID: \"0bcd7af3-c2f8-48a2-ae11-a7cce16f429e\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-2300af-64b4d74fff-q2xp7"
Nov 03 18:37:20.059825 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:20.059708    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzf6n\" (UniqueName: \"kubernetes.io/projected/0bcd7af3-c2f8-48a2-ae11-a7cce16f429e-kube-api-access-hzf6n\") pod \"payload-processing-e2e-authcfg-http-2300af-64b4d74fff-q2xp7\" (UID: \"0bcd7af3-c2f8-48a2-ae11-a7cce16f429e\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-2300af-64b4d74fff-q2xp7"
Nov 03 18:37:20.081009 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:20.080972    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-2300af\""
Nov 03 18:37:20.081199 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:20.081056    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-2300af-dockercfg-qv9dr\""
Nov 03 18:37:20.160788 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:20.160744    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzf6n\" (UniqueName: \"kubernetes.io/projected/0bcd7af3-c2f8-48a2-ae11-a7cce16f429e-kube-api-access-hzf6n\") pod \"payload-processing-e2e-authcfg-http-2300af-64b4d74fff-q2xp7\" (UID: \"0bcd7af3-c2f8-48a2-ae11-a7cce16f429e\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-2300af-64b4d74fff-q2xp7"
Nov 03 18:37:20.161282 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:20.161249    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bcd7af3-c2f8-48a2-ae11-a7cce16f429e-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-2300af-64b4d74fff-q2xp7\" (UID: \"0bcd7af3-c2f8-48a2-ae11-a7cce16f429e\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-2300af-64b4d74fff-q2xp7"
Nov 03 18:37:20.162013 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:20.161987    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bcd7af3-c2f8-48a2-ae11-a7cce16f429e-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-2300af-64b4d74fff-q2xp7\" (UID: \"0bcd7af3-c2f8-48a2-ae11-a7cce16f429e\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-2300af-64b4d74fff-q2xp7"
Nov 03 18:37:20.218373 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:20.218337    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzf6n\" (UniqueName: \"kubernetes.io/projected/0bcd7af3-c2f8-48a2-ae11-a7cce16f429e-kube-api-access-hzf6n\") pod \"payload-processing-e2e-authcfg-http-2300af-64b4d74fff-q2xp7\" (UID: \"0bcd7af3-c2f8-48a2-ae11-a7cce16f429e\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-2300af-64b4d74fff-q2xp7"
Nov 03 18:37:20.303117 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:20.303087    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-2300af-64b4d74fff-q2xp7"
Nov 03 18:37:20.670970 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:20.670886    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-2300af-64b4d74fff-q2xp7"]
Nov 03 18:37:20.677770 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:37:20.677738    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bcd7af3_c2f8_48a2_ae11_a7cce16f429e.slice/crio-e1c48d75ada9c3bf9fede9a4482cf2e42b7f170be5747bc36c2bd58815527511 WatchSource:0}: Error finding container e1c48d75ada9c3bf9fede9a4482cf2e42b7f170be5747bc36c2bd58815527511: Status 404 returned error can't find the container with id e1c48d75ada9c3bf9fede9a4482cf2e42b7f170be5747bc36c2bd58815527511
Nov 03 18:37:21.305532 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:21.303257    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-2300af-64b4d74fff-q2xp7" event={"ID":"0bcd7af3-c2f8-48a2-ae11-a7cce16f429e","Type":"ContainerStarted","Data":"52af7af3416d12b049172d9fc646fb5d4c5144069b2764489e0f1311bcfe2458"}
Nov 03 18:37:21.305532 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:21.303329    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-2300af-64b4d74fff-q2xp7" event={"ID":"0bcd7af3-c2f8-48a2-ae11-a7cce16f429e","Type":"ContainerStarted","Data":"e1c48d75ada9c3bf9fede9a4482cf2e42b7f170be5747bc36c2bd58815527511"}
Nov 03 18:37:21.305532 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:21.303497    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-2300af-64b4d74fff-q2xp7"
Nov 03 18:37:21.411237 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:21.411185    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-2300af-64b4d74fff-q2xp7" podStartSLOduration=2.411172483 podStartE2EDuration="2.411172483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:37:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:21.41076788 +0000 UTC m=+3080.426965948" watchObservedRunningTime="2026-11-03 18:37:21.411172483 +0000 UTC m=+3080.427370552"
Nov 03 18:37:24.577332 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:37:24.577290    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:37:32.310949 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:37:32.310916    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-2300af-64b4d74fff-q2xp7"
Nov 03 18:37:54.576841 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:37:54.576799    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:38:03.044685 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:38:03.044644    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73f4ae16-nkwrx"]
Nov 03 18:38:03.090667 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:38:03.090626    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73f4ae16-nkwrx"]
Nov 03 18:38:03.103112 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:38:03.103078    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b52a2b0-d10c-4d06-9f78-e318f6a02973" path="/var/lib/kubelet/pods/9b52a2b0-d10c-4d06-9f78-e318f6a02973/volumes"
Nov 03 18:38:06.488323 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:38:06.488281    5315 scope.go:122] "RemoveContainer" containerID="4cb9accc3c1b0238de3cb32a9c6be820aacc07f5e27eecbcbb1fb2c4c3c86086"
Nov 03 18:38:24.577569 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:38:24.577528    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:38:54.577512 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:38:54.577420    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:39:01.877825 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:01.877788    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2300af-8njn2"]
Nov 03 18:39:01.878351 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:01.878154    5315 cpu_manager.go:414] "containerMap: removing container" podUID="2fcbb83b-f0fc-427b-806c-62cf4fa66437" containerName="istio-proxy"
Nov 03 18:39:01.878351 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:01.878165    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fcbb83b-f0fc-427b-806c-62cf4fa66437" containerName="istio-proxy"
Nov 03 18:39:01.878351 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:01.878242    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fcbb83b-f0fc-427b-806c-62cf4fa66437" containerName="istio-proxy"
Nov 03 18:39:01.881397 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:01.881374    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2300af-8njn2"
Nov 03 18:39:01.934398 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:01.934369    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:39:01.984855 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:01.984817    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:39:01.984855 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:01.984816    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:39:01.985071 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:01.984820    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgm5\""
Nov 03 18:39:01.986180 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:01.986160    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2300af-8njn2"]
Nov 03 18:39:02.015342 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:02.015307    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5h48\" (UniqueName: \"kubernetes.io/projected/fb5492d2-d328-4fe6-9300-b1ea72a97b06-kube-api-access-q5h48\") pod \"maas-api-revoke-keys-e2e-authcfg-http-2300af-8njn2\" (UID: \"fb5492d2-d328-4fe6-9300-b1ea72a97b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2300af-8njn2"
Nov 03 18:39:02.015504 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:02.015384    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb5492d2-d328-4fe6-9300-b1ea72a97b06-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-2300af-8njn2\" (UID: \"fb5492d2-d328-4fe6-9300-b1ea72a97b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2300af-8njn2"
Nov 03 18:39:02.015504 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:02.015433    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb5492d2-d328-4fe6-9300-b1ea72a97b06-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-2300af-8njn2\" (UID: \"fb5492d2-d328-4fe6-9300-b1ea72a97b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2300af-8njn2"
Nov 03 18:39:02.116400 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:02.116315    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5h48\" (UniqueName: \"kubernetes.io/projected/fb5492d2-d328-4fe6-9300-b1ea72a97b06-kube-api-access-q5h48\") pod \"maas-api-revoke-keys-e2e-authcfg-http-2300af-8njn2\" (UID: \"fb5492d2-d328-4fe6-9300-b1ea72a97b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2300af-8njn2"
Nov 03 18:39:02.116590 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:02.116462    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb5492d2-d328-4fe6-9300-b1ea72a97b06-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-2300af-8njn2\" (UID: \"fb5492d2-d328-4fe6-9300-b1ea72a97b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2300af-8njn2"
Nov 03 18:39:02.116590 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:02.116516    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb5492d2-d328-4fe6-9300-b1ea72a97b06-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-2300af-8njn2\" (UID: \"fb5492d2-d328-4fe6-9300-b1ea72a97b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2300af-8njn2"
Nov 03 18:39:02.117183 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:02.117158    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb5492d2-d328-4fe6-9300-b1ea72a97b06-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-2300af-8njn2\" (UID: \"fb5492d2-d328-4fe6-9300-b1ea72a97b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2300af-8njn2"
Nov 03 18:39:02.119388 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:02.119364    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb5492d2-d328-4fe6-9300-b1ea72a97b06-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-2300af-8njn2\" (UID: \"fb5492d2-d328-4fe6-9300-b1ea72a97b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2300af-8njn2"
Nov 03 18:39:02.178123 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:02.178044    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5h48\" (UniqueName: \"kubernetes.io/projected/fb5492d2-d328-4fe6-9300-b1ea72a97b06-kube-api-access-q5h48\") pod \"maas-api-revoke-keys-e2e-authcfg-http-2300af-8njn2\" (UID: \"fb5492d2-d328-4fe6-9300-b1ea72a97b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2300af-8njn2"
Nov 03 18:39:02.197179 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:02.197136    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2300af-8njn2"
Nov 03 18:39:02.568804 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:02.568772    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2300af-8njn2"]
Nov 03 18:39:02.571705 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:39:02.571665    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb5492d2_d328_4fe6_9300_b1ea72a97b06.slice/crio-1c0351421b53734e33e716a7a1c415915d482ed74d36c9f7861b1a09e5d367ba WatchSource:0}: Error finding container 1c0351421b53734e33e716a7a1c415915d482ed74d36c9f7861b1a09e5d367ba: Status 404 returned error can't find the container with id 1c0351421b53734e33e716a7a1c415915d482ed74d36c9f7861b1a09e5d367ba
Nov 03 18:39:02.787150 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:02.787114    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2300af-8njn2" event={"ID":"fb5492d2-d328-4fe6-9300-b1ea72a97b06","Type":"ContainerStarted","Data":"1c0351421b53734e33e716a7a1c415915d482ed74d36c9f7861b1a09e5d367ba"}
Nov 03 18:39:03.793214 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:03.793181    5315 generic.go:356] "Generic (PLEG): container finished" podID="fb5492d2-d328-4fe6-9300-b1ea72a97b06" containerID="ccf1c54f0e1b2546415081a71a5626ad6396fdb674ab7b3080a8a0bf5806c045" exitCode=0
Nov 03 18:39:03.793639 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:03.793276    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2300af-8njn2" event={"ID":"fb5492d2-d328-4fe6-9300-b1ea72a97b06","Type":"ContainerDied","Data":"ccf1c54f0e1b2546415081a71a5626ad6396fdb674ab7b3080a8a0bf5806c045"}
Nov 03 18:39:04.920802 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:04.920775    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2300af-8njn2"
Nov 03 18:39:05.042178 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:05.042143    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb5492d2-d328-4fe6-9300-b1ea72a97b06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb5492d2-d328-4fe6-9300-b1ea72a97b06-maas-api-xks-ca\") pod \"fb5492d2-d328-4fe6-9300-b1ea72a97b06\" (UID: \"fb5492d2-d328-4fe6-9300-b1ea72a97b06\") "
Nov 03 18:39:05.042373 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:05.042237    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb5492d2-d328-4fe6-9300-b1ea72a97b06-kube-api-access-q5h48\" (UniqueName: \"kubernetes.io/projected/fb5492d2-d328-4fe6-9300-b1ea72a97b06-kube-api-access-q5h48\") pod \"fb5492d2-d328-4fe6-9300-b1ea72a97b06\" (UID: \"fb5492d2-d328-4fe6-9300-b1ea72a97b06\") "
Nov 03 18:39:05.042373 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:05.042336    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb5492d2-d328-4fe6-9300-b1ea72a97b06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb5492d2-d328-4fe6-9300-b1ea72a97b06-maas-api-openshift-ca\") pod \"fb5492d2-d328-4fe6-9300-b1ea72a97b06\" (UID: \"fb5492d2-d328-4fe6-9300-b1ea72a97b06\") "
Nov 03 18:39:05.042800 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:05.042758    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb5492d2-d328-4fe6-9300-b1ea72a97b06-maas-api-openshift-ca" pod "fb5492d2-d328-4fe6-9300-b1ea72a97b06" (UID: "fb5492d2-d328-4fe6-9300-b1ea72a97b06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:39:05.044496 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:05.044471    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb5492d2-d328-4fe6-9300-b1ea72a97b06-maas-api-xks-ca" pod "fb5492d2-d328-4fe6-9300-b1ea72a97b06" (UID: "fb5492d2-d328-4fe6-9300-b1ea72a97b06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:39:05.044648 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:05.044556    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb5492d2-d328-4fe6-9300-b1ea72a97b06-kube-api-access-q5h48" pod "fb5492d2-d328-4fe6-9300-b1ea72a97b06" (UID: "fb5492d2-d328-4fe6-9300-b1ea72a97b06"). InnerVolumeSpecName "kube-api-access-q5h48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:39:05.143098 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:05.143066    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb5492d2-d328-4fe6-9300-b1ea72a97b06-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:39:05.143098 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:05.143094    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb5492d2-d328-4fe6-9300-b1ea72a97b06-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:39:05.143098 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:05.143105    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5h48\" (UniqueName: \"kubernetes.io/projected/fb5492d2-d328-4fe6-9300-b1ea72a97b06-kube-api-access-q5h48\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:39:05.803522 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:05.803495    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2300af-8njn2"
Nov 03 18:39:05.803740 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:05.803536    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2300af-8njn2" event={"ID":"fb5492d2-d328-4fe6-9300-b1ea72a97b06","Type":"ContainerDied","Data":"1c0351421b53734e33e716a7a1c415915d482ed74d36c9f7861b1a09e5d367ba"}
Nov 03 18:39:05.803740 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:05.803584    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c0351421b53734e33e716a7a1c415915d482ed74d36c9f7861b1a09e5d367ba"
Nov 03 18:39:06.513654 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:06.513623    5315 scope.go:122] "RemoveContainer" containerID="e96775147cfc8e6df0ad4308917e212cbe42f9b68de2cbaa209cffe20e2467b7"
Nov 03 18:39:13.418813 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:13.418777    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-2300af-64b4d74fff-q2xp7"]
Nov 03 18:39:18.419236 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:18.419185    5315 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-2300af-64b4d74fff-q2xp7" podUID="0bcd7af3-c2f8-48a2-ae11-a7cce16f429e" containerName="payload-processing" containerID="cri-o://52af7af3416d12b049172d9fc646fb5d4c5144069b2764489e0f1311bcfe2458" gracePeriod=25
Nov 03 18:39:18.660284 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:18.660251    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-2300af-64b4d74fff-q2xp7"
Nov 03 18:39:18.684238 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:18.684162    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bcd7af3-c2f8-48a2-ae11-a7cce16f429e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bcd7af3-c2f8-48a2-ae11-a7cce16f429e-plugins-config-volume\") pod \"0bcd7af3-c2f8-48a2-ae11-a7cce16f429e\" (UID: \"0bcd7af3-c2f8-48a2-ae11-a7cce16f429e\") "
Nov 03 18:39:18.684238 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:18.684209    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bcd7af3-c2f8-48a2-ae11-a7cce16f429e-kube-api-access-hzf6n\" (UniqueName: \"kubernetes.io/projected/0bcd7af3-c2f8-48a2-ae11-a7cce16f429e-kube-api-access-hzf6n\") pod \"0bcd7af3-c2f8-48a2-ae11-a7cce16f429e\" (UID: \"0bcd7af3-c2f8-48a2-ae11-a7cce16f429e\") "
Nov 03 18:39:18.684515 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:18.684494    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bcd7af3-c2f8-48a2-ae11-a7cce16f429e-plugins-config-volume" pod "0bcd7af3-c2f8-48a2-ae11-a7cce16f429e" (UID: "0bcd7af3-c2f8-48a2-ae11-a7cce16f429e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:39:18.686299 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:18.686273    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bcd7af3-c2f8-48a2-ae11-a7cce16f429e-kube-api-access-hzf6n" pod "0bcd7af3-c2f8-48a2-ae11-a7cce16f429e" (UID: "0bcd7af3-c2f8-48a2-ae11-a7cce16f429e"). InnerVolumeSpecName "kube-api-access-hzf6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:39:18.785043 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:18.785005    5315 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bcd7af3-c2f8-48a2-ae11-a7cce16f429e-plugins-config-volume\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:39:18.785043 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:18.785037    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzf6n\" (UniqueName: \"kubernetes.io/projected/0bcd7af3-c2f8-48a2-ae11-a7cce16f429e-kube-api-access-hzf6n\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:39:18.920671 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:18.920628    5315 generic.go:356] "Generic (PLEG): container finished" podID="0bcd7af3-c2f8-48a2-ae11-a7cce16f429e" containerID="52af7af3416d12b049172d9fc646fb5d4c5144069b2764489e0f1311bcfe2458" exitCode=0
Nov 03 18:39:18.920845 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:18.920683    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-2300af-64b4d74fff-q2xp7"
Nov 03 18:39:18.920845 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:18.920725    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-2300af-64b4d74fff-q2xp7" event={"ID":"0bcd7af3-c2f8-48a2-ae11-a7cce16f429e","Type":"ContainerDied","Data":"52af7af3416d12b049172d9fc646fb5d4c5144069b2764489e0f1311bcfe2458"}
Nov 03 18:39:18.920845 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:18.920794    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-2300af-64b4d74fff-q2xp7" event={"ID":"0bcd7af3-c2f8-48a2-ae11-a7cce16f429e","Type":"ContainerDied","Data":"e1c48d75ada9c3bf9fede9a4482cf2e42b7f170be5747bc36c2bd58815527511"}
Nov 03 18:39:18.920845 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:18.920827    5315 scope.go:122] "RemoveContainer" containerID="52af7af3416d12b049172d9fc646fb5d4c5144069b2764489e0f1311bcfe2458"
Nov 03 18:39:18.930196 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:18.930174    5315 scope.go:122] "RemoveContainer" containerID="52af7af3416d12b049172d9fc646fb5d4c5144069b2764489e0f1311bcfe2458"
Nov 03 18:39:18.930512 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:39:18.930488    5315 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52af7af3416d12b049172d9fc646fb5d4c5144069b2764489e0f1311bcfe2458\": container with ID starting with 52af7af3416d12b049172d9fc646fb5d4c5144069b2764489e0f1311bcfe2458 not found: ID does not exist" containerID="52af7af3416d12b049172d9fc646fb5d4c5144069b2764489e0f1311bcfe2458"
Nov 03 18:39:18.930570 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:18.930526    5315 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52af7af3416d12b049172d9fc646fb5d4c5144069b2764489e0f1311bcfe2458"} err="failed to get container status \"52af7af3416d12b049172d9fc646fb5d4c5144069b2764489e0f1311bcfe2458\": rpc error: code = NotFound desc = could not find container \"52af7af3416d12b049172d9fc646fb5d4c5144069b2764489e0f1311bcfe2458\": container with ID starting with 52af7af3416d12b049172d9fc646fb5d4c5144069b2764489e0f1311bcfe2458 not found: ID does not exist"
Nov 03 18:39:19.088750 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:19.088717    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-2300af-64b4d74fff-q2xp7"]
Nov 03 18:39:19.095158 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:19.095126    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-2300af-64b4d74fff-q2xp7"]
Nov 03 18:39:19.102136 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:19.102109    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bcd7af3-c2f8-48a2-ae11-a7cce16f429e" path="/var/lib/kubelet/pods/0bcd7af3-c2f8-48a2-ae11-a7cce16f429e/volumes"
Nov 03 18:39:24.594448 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:39:24.594407    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:39:26.643102 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:26.643066    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2300af-8njn2"]
Nov 03 18:39:26.693323 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:26.693290    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2300af-8njn2"]
Nov 03 18:39:27.103004 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:27.102970    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb5492d2-d328-4fe6-9300-b1ea72a97b06" path="/var/lib/kubelet/pods/fb5492d2-d328-4fe6-9300-b1ea72a97b06/volumes"
Nov 03 18:39:51.053232 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:51.053192    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-vz2p7"]
Nov 03 18:39:51.132272 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:51.132226    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-vz2p7"]
Nov 03 18:39:51.179916 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:39:51.179868    5315 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-452b6d7b-vz2p7\" is forbidden: User \"system:node:ip-10-0-128-204.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-128-204.us-west-2.compute.internal' and this object" podUID="bf6a4396-a6ee-4756-a602-a15156edc081" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-vz2p7"
Nov 03 18:39:53.103014 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:53.102976    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf6a4396-a6ee-4756-a602-a15156edc081" path="/var/lib/kubelet/pods/bf6a4396-a6ee-4756-a602-a15156edc081/volumes"
Nov 03 18:39:54.576297 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:39:54.576254    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:39:57.881773 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:57.881723    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-bn99v"]
Nov 03 18:39:57.882306 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:57.882153    5315 cpu_manager.go:414] "containerMap: removing container" podUID="0bcd7af3-c2f8-48a2-ae11-a7cce16f429e" containerName="payload-processing"
Nov 03 18:39:57.882306 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:57.882168    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bcd7af3-c2f8-48a2-ae11-a7cce16f429e" containerName="payload-processing"
Nov 03 18:39:57.882306 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:57.882180    5315 cpu_manager.go:414] "containerMap: removing container" podUID="fb5492d2-d328-4fe6-9300-b1ea72a97b06" containerName="revoke-keys"
Nov 03 18:39:57.882306 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:57.882185    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb5492d2-d328-4fe6-9300-b1ea72a97b06" containerName="revoke-keys"
Nov 03 18:39:57.882306 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:57.882257    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bcd7af3-c2f8-48a2-ae11-a7cce16f429e" containerName="payload-processing"
Nov 03 18:39:57.882306 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:57.882265    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb5492d2-d328-4fe6-9300-b1ea72a97b06" containerName="revoke-keys"
Nov 03 18:39:57.885645 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:57.885602    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-bn99v"
Nov 03 18:39:57.938674 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:57.938642    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:39:57.954387 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:57.954351    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcdbs\" (UniqueName: \"kubernetes.io/projected/64a9cfd3-025c-4ddf-9740-8516f8f091eb-kube-api-access-jcdbs\") pod \"maas-api-revoke-sub-f03629db-bn99v\" (UID: \"64a9cfd3-025c-4ddf-9740-8516f8f091eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-bn99v"
Nov 03 18:39:57.954535 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:57.954400    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64a9cfd3-025c-4ddf-9740-8516f8f091eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-bn99v\" (UID: \"64a9cfd3-025c-4ddf-9740-8516f8f091eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-bn99v"
Nov 03 18:39:57.954592 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:57.954566    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64a9cfd3-025c-4ddf-9740-8516f8f091eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-bn99v\" (UID: \"64a9cfd3-025c-4ddf-9740-8516f8f091eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-bn99v"
Nov 03 18:39:57.989072 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:57.989034    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:39:57.989251 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:57.989084    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:39:57.989251 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:57.989157    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgm5\""
Nov 03 18:39:57.990446 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:57.990426    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-bn99v"]
Nov 03 18:39:58.055074 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:58.055039    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64a9cfd3-025c-4ddf-9740-8516f8f091eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-bn99v\" (UID: \"64a9cfd3-025c-4ddf-9740-8516f8f091eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-bn99v"
Nov 03 18:39:58.055264 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:58.055169    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcdbs\" (UniqueName: \"kubernetes.io/projected/64a9cfd3-025c-4ddf-9740-8516f8f091eb-kube-api-access-jcdbs\") pod \"maas-api-revoke-sub-f03629db-bn99v\" (UID: \"64a9cfd3-025c-4ddf-9740-8516f8f091eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-bn99v"
Nov 03 18:39:58.055264 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:58.055193    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64a9cfd3-025c-4ddf-9740-8516f8f091eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-bn99v\" (UID: \"64a9cfd3-025c-4ddf-9740-8516f8f091eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-bn99v"
Nov 03 18:39:58.055890 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:58.055860    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64a9cfd3-025c-4ddf-9740-8516f8f091eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-bn99v\" (UID: \"64a9cfd3-025c-4ddf-9740-8516f8f091eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-bn99v"
Nov 03 18:39:58.057511 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:58.057480    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64a9cfd3-025c-4ddf-9740-8516f8f091eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-bn99v\" (UID: \"64a9cfd3-025c-4ddf-9740-8516f8f091eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-bn99v"
Nov 03 18:39:58.117417 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:58.117388    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcdbs\" (UniqueName: \"kubernetes.io/projected/64a9cfd3-025c-4ddf-9740-8516f8f091eb-kube-api-access-jcdbs\") pod \"maas-api-revoke-sub-f03629db-bn99v\" (UID: \"64a9cfd3-025c-4ddf-9740-8516f8f091eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-bn99v"
Nov 03 18:39:58.198278 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:58.198183    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-bn99v"
Nov 03 18:39:58.570038 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:58.570014    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-bn99v"]
Nov 03 18:39:58.572853 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:39:58.572821    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64a9cfd3_025c_4ddf_9740_8516f8f091eb.slice/crio-106869ba1bf3eae0f32b133911e76cb91d3dc91e9dbc71461ce657cd2356671b WatchSource:0}: Error finding container 106869ba1bf3eae0f32b133911e76cb91d3dc91e9dbc71461ce657cd2356671b: Status 404 returned error can't find the container with id 106869ba1bf3eae0f32b133911e76cb91d3dc91e9dbc71461ce657cd2356671b
Nov 03 18:39:59.159083 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:59.158988    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-bn99v" event={"ID":"64a9cfd3-025c-4ddf-9740-8516f8f091eb","Type":"ContainerStarted","Data":"91fd9efba502fe4795c75c90e87047cfccf663a1bfe52903a8329b87a2fccbd1"}
Nov 03 18:39:59.159083 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:59.159046    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-bn99v" event={"ID":"64a9cfd3-025c-4ddf-9740-8516f8f091eb","Type":"ContainerStarted","Data":"106869ba1bf3eae0f32b133911e76cb91d3dc91e9dbc71461ce657cd2356671b"}
Nov 03 18:39:59.275758 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:39:59.275699    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-bn99v" podStartSLOduration=2.275686225 podStartE2EDuration="2.275686225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:39:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:39:59.274701844 +0000 UTC m=+3238.290899914" watchObservedRunningTime="2026-11-03 18:39:59.275686225 +0000 UTC m=+3238.291884290"
Nov 03 18:40:00.164856 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:00.164825    5315 generic.go:356] "Generic (PLEG): container finished" podID="64a9cfd3-025c-4ddf-9740-8516f8f091eb" containerID="91fd9efba502fe4795c75c90e87047cfccf663a1bfe52903a8329b87a2fccbd1" exitCode=0
Nov 03 18:40:00.165334 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:00.164917    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-bn99v" event={"ID":"64a9cfd3-025c-4ddf-9740-8516f8f091eb","Type":"ContainerDied","Data":"91fd9efba502fe4795c75c90e87047cfccf663a1bfe52903a8329b87a2fccbd1"}
Nov 03 18:40:01.305123 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:01.305093    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-bn99v"
Nov 03 18:40:01.388037 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:01.387996    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64a9cfd3-025c-4ddf-9740-8516f8f091eb-kube-api-access-jcdbs\" (UniqueName: \"kubernetes.io/projected/64a9cfd3-025c-4ddf-9740-8516f8f091eb-kube-api-access-jcdbs\") pod \"64a9cfd3-025c-4ddf-9740-8516f8f091eb\" (UID: \"64a9cfd3-025c-4ddf-9740-8516f8f091eb\") "
Nov 03 18:40:01.388244 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:01.388102    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64a9cfd3-025c-4ddf-9740-8516f8f091eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64a9cfd3-025c-4ddf-9740-8516f8f091eb-maas-api-xks-ca\") pod \"64a9cfd3-025c-4ddf-9740-8516f8f091eb\" (UID: \"64a9cfd3-025c-4ddf-9740-8516f8f091eb\") "
Nov 03 18:40:01.388244 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:01.388168    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64a9cfd3-025c-4ddf-9740-8516f8f091eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64a9cfd3-025c-4ddf-9740-8516f8f091eb-maas-api-openshift-ca\") pod \"64a9cfd3-025c-4ddf-9740-8516f8f091eb\" (UID: \"64a9cfd3-025c-4ddf-9740-8516f8f091eb\") "
Nov 03 18:40:01.388484 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:01.388457    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64a9cfd3-025c-4ddf-9740-8516f8f091eb-maas-api-openshift-ca" pod "64a9cfd3-025c-4ddf-9740-8516f8f091eb" (UID: "64a9cfd3-025c-4ddf-9740-8516f8f091eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:40:01.390104 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:01.390081    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64a9cfd3-025c-4ddf-9740-8516f8f091eb-kube-api-access-jcdbs" pod "64a9cfd3-025c-4ddf-9740-8516f8f091eb" (UID: "64a9cfd3-025c-4ddf-9740-8516f8f091eb"). InnerVolumeSpecName "kube-api-access-jcdbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:40:01.390185 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:01.390166    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64a9cfd3-025c-4ddf-9740-8516f8f091eb-maas-api-xks-ca" pod "64a9cfd3-025c-4ddf-9740-8516f8f091eb" (UID: "64a9cfd3-025c-4ddf-9740-8516f8f091eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:40:01.489426 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:01.489340    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcdbs\" (UniqueName: \"kubernetes.io/projected/64a9cfd3-025c-4ddf-9740-8516f8f091eb-kube-api-access-jcdbs\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:40:01.489426 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:01.489376    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64a9cfd3-025c-4ddf-9740-8516f8f091eb-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:40:01.489426 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:01.489386    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64a9cfd3-025c-4ddf-9740-8516f8f091eb-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:40:02.176194 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:02.176157    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-bn99v"
Nov 03 18:40:02.176395 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:02.176157    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-bn99v" event={"ID":"64a9cfd3-025c-4ddf-9740-8516f8f091eb","Type":"ContainerDied","Data":"106869ba1bf3eae0f32b133911e76cb91d3dc91e9dbc71461ce657cd2356671b"}
Nov 03 18:40:02.176395 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:02.176293    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="106869ba1bf3eae0f32b133911e76cb91d3dc91e9dbc71461ce657cd2356671b"
Nov 03 18:40:06.533013 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:06.532972    5315 scope.go:122] "RemoveContainer" containerID="d6c2d8f0abbe6cd01ccc52b1083465e59413b6b765a05c1ae4b226ce3861c51b"
Nov 03 18:40:18.972858 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:18.972820    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjdmn"]
Nov 03 18:40:18.973341 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:18.973275    5315 cpu_manager.go:414] "containerMap: removing container" podUID="64a9cfd3-025c-4ddf-9740-8516f8f091eb" containerName="revoke-keys"
Nov 03 18:40:18.973341 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:18.973292    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64a9cfd3-025c-4ddf-9740-8516f8f091eb" containerName="revoke-keys"
Nov 03 18:40:18.973464 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:18.973376    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="64a9cfd3-025c-4ddf-9740-8516f8f091eb" containerName="revoke-keys"
Nov 03 18:40:18.977868 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:18.977845    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjdmn"
Nov 03 18:40:19.029340 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:19.029295    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67fa6bc6-22a9-4230-a466-0a6b3929b522-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-fjdmn\" (UID: \"67fa6bc6-22a9-4230-a466-0a6b3929b522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjdmn"
Nov 03 18:40:19.029528 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:19.029365    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67fa6bc6-22a9-4230-a466-0a6b3929b522-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-fjdmn\" (UID: \"67fa6bc6-22a9-4230-a466-0a6b3929b522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjdmn"
Nov 03 18:40:19.029528 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:19.029452    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72cxk\" (UniqueName: \"kubernetes.io/projected/67fa6bc6-22a9-4230-a466-0a6b3929b522-kube-api-access-72cxk\") pod \"maas-api-revoke-sub-0165d7d2-fjdmn\" (UID: \"67fa6bc6-22a9-4230-a466-0a6b3929b522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjdmn"
Nov 03 18:40:19.081548 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:19.081516    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:40:19.081548 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:19.081520    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:40:19.081548 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:19.081519    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:40:19.081823 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:19.081564    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgm5\""
Nov 03 18:40:19.082980 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:19.082961    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjdmn"]
Nov 03 18:40:19.130259 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:19.130223    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67fa6bc6-22a9-4230-a466-0a6b3929b522-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-fjdmn\" (UID: \"67fa6bc6-22a9-4230-a466-0a6b3929b522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjdmn"
Nov 03 18:40:19.130493 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:19.130269    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67fa6bc6-22a9-4230-a466-0a6b3929b522-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-fjdmn\" (UID: \"67fa6bc6-22a9-4230-a466-0a6b3929b522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjdmn"
Nov 03 18:40:19.130493 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:19.130315    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72cxk\" (UniqueName: \"kubernetes.io/projected/67fa6bc6-22a9-4230-a466-0a6b3929b522-kube-api-access-72cxk\") pod \"maas-api-revoke-sub-0165d7d2-fjdmn\" (UID: \"67fa6bc6-22a9-4230-a466-0a6b3929b522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjdmn"
Nov 03 18:40:19.130977 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:19.130947    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67fa6bc6-22a9-4230-a466-0a6b3929b522-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-fjdmn\" (UID: \"67fa6bc6-22a9-4230-a466-0a6b3929b522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjdmn"
Nov 03 18:40:19.132504 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:19.132478    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67fa6bc6-22a9-4230-a466-0a6b3929b522-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-fjdmn\" (UID: \"67fa6bc6-22a9-4230-a466-0a6b3929b522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjdmn"
Nov 03 18:40:19.191632 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:19.191573    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72cxk\" (UniqueName: \"kubernetes.io/projected/67fa6bc6-22a9-4230-a466-0a6b3929b522-kube-api-access-72cxk\") pod \"maas-api-revoke-sub-0165d7d2-fjdmn\" (UID: \"67fa6bc6-22a9-4230-a466-0a6b3929b522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjdmn"
Nov 03 18:40:19.290051 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:19.290021    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjdmn"
Nov 03 18:40:19.661130 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:19.661100    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjdmn"]
Nov 03 18:40:19.663357 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:40:19.663328    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67fa6bc6_22a9_4230_a466_0a6b3929b522.slice/crio-d0ed71aee822ae69998412ecdd0f6f22cf5765eb6851b88dfdc702db0995ec56 WatchSource:0}: Error finding container d0ed71aee822ae69998412ecdd0f6f22cf5765eb6851b88dfdc702db0995ec56: Status 404 returned error can't find the container with id d0ed71aee822ae69998412ecdd0f6f22cf5765eb6851b88dfdc702db0995ec56
Nov 03 18:40:20.281943 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:20.281908    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjdmn" event={"ID":"67fa6bc6-22a9-4230-a466-0a6b3929b522","Type":"ContainerStarted","Data":"7d7e60c2da24b6e537d9993c3d6ff55685fec37aead59a6b49c12ef7089d211b"}
Nov 03 18:40:20.282359 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:20.281975    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjdmn" event={"ID":"67fa6bc6-22a9-4230-a466-0a6b3929b522","Type":"ContainerStarted","Data":"d0ed71aee822ae69998412ecdd0f6f22cf5765eb6851b88dfdc702db0995ec56"}
Nov 03 18:40:20.399314 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:20.399250    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjdmn" podStartSLOduration=2.399237688 podStartE2EDuration="2.399237688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:40:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:40:20.39772791 +0000 UTC m=+3259.413925980" watchObservedRunningTime="2026-11-03 18:40:20.399237688 +0000 UTC m=+3259.415435800"
Nov 03 18:40:21.287903 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:21.287860    5315 generic.go:356] "Generic (PLEG): container finished" podID="67fa6bc6-22a9-4230-a466-0a6b3929b522" containerID="7d7e60c2da24b6e537d9993c3d6ff55685fec37aead59a6b49c12ef7089d211b" exitCode=0
Nov 03 18:40:21.288326 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:21.287911    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjdmn" event={"ID":"67fa6bc6-22a9-4230-a466-0a6b3929b522","Type":"ContainerDied","Data":"7d7e60c2da24b6e537d9993c3d6ff55685fec37aead59a6b49c12ef7089d211b"}
Nov 03 18:40:22.430321 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:22.430294    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjdmn"
Nov 03 18:40:22.457515 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:22.457480    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67fa6bc6-22a9-4230-a466-0a6b3929b522-kube-api-access-72cxk\" (UniqueName: \"kubernetes.io/projected/67fa6bc6-22a9-4230-a466-0a6b3929b522-kube-api-access-72cxk\") pod \"67fa6bc6-22a9-4230-a466-0a6b3929b522\" (UID: \"67fa6bc6-22a9-4230-a466-0a6b3929b522\") "
Nov 03 18:40:22.457705 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:22.457548    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67fa6bc6-22a9-4230-a466-0a6b3929b522-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67fa6bc6-22a9-4230-a466-0a6b3929b522-maas-api-openshift-ca\") pod \"67fa6bc6-22a9-4230-a466-0a6b3929b522\" (UID: \"67fa6bc6-22a9-4230-a466-0a6b3929b522\") "
Nov 03 18:40:22.457705 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:22.457582    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67fa6bc6-22a9-4230-a466-0a6b3929b522-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67fa6bc6-22a9-4230-a466-0a6b3929b522-maas-api-xks-ca\") pod \"67fa6bc6-22a9-4230-a466-0a6b3929b522\" (UID: \"67fa6bc6-22a9-4230-a466-0a6b3929b522\") "
Nov 03 18:40:22.457987 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:22.457955    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67fa6bc6-22a9-4230-a466-0a6b3929b522-maas-api-openshift-ca" pod "67fa6bc6-22a9-4230-a466-0a6b3929b522" (UID: "67fa6bc6-22a9-4230-a466-0a6b3929b522"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:40:22.459555 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:22.459521    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67fa6bc6-22a9-4230-a466-0a6b3929b522-kube-api-access-72cxk" pod "67fa6bc6-22a9-4230-a466-0a6b3929b522" (UID: "67fa6bc6-22a9-4230-a466-0a6b3929b522"). InnerVolumeSpecName "kube-api-access-72cxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:40:22.459555 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:22.459542    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67fa6bc6-22a9-4230-a466-0a6b3929b522-maas-api-xks-ca" pod "67fa6bc6-22a9-4230-a466-0a6b3929b522" (UID: "67fa6bc6-22a9-4230-a466-0a6b3929b522"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:40:22.558757 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:22.558673    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67fa6bc6-22a9-4230-a466-0a6b3929b522-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:40:22.558757 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:22.558706    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72cxk\" (UniqueName: \"kubernetes.io/projected/67fa6bc6-22a9-4230-a466-0a6b3929b522-kube-api-access-72cxk\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:40:22.558757 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:22.558717    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67fa6bc6-22a9-4230-a466-0a6b3929b522-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:40:23.297717 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:23.297684    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjdmn"
Nov 03 18:40:23.297717 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:23.297693    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjdmn" event={"ID":"67fa6bc6-22a9-4230-a466-0a6b3929b522","Type":"ContainerDied","Data":"d0ed71aee822ae69998412ecdd0f6f22cf5765eb6851b88dfdc702db0995ec56"}
Nov 03 18:40:23.297934 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:23.297746    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0ed71aee822ae69998412ecdd0f6f22cf5765eb6851b88dfdc702db0995ec56"
Nov 03 18:40:24.577116 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:40:24.577072    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:40:26.049438 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:26.049402    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-b5j78"]
Nov 03 18:40:26.099811 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:26.099774    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-b5j78"]
Nov 03 18:40:27.101914 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:27.101880    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9108b98c-1c45-49d5-9c62-3a6272b39cad" path="/var/lib/kubelet/pods/9108b98c-1c45-49d5-9c62-3a6272b39cad/volumes"
Nov 03 18:40:30.758166 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:30.758130    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-297n8"]
Nov 03 18:40:30.758572 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:30.758501    5315 cpu_manager.go:414] "containerMap: removing container" podUID="67fa6bc6-22a9-4230-a466-0a6b3929b522" containerName="revoke-keys"
Nov 03 18:40:30.758572 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:30.758512    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67fa6bc6-22a9-4230-a466-0a6b3929b522" containerName="revoke-keys"
Nov 03 18:40:30.758572 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:30.758565    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="67fa6bc6-22a9-4230-a466-0a6b3929b522" containerName="revoke-keys"
Nov 03 18:40:30.762819 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:30.762796    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-297n8"
Nov 03 18:40:30.831563 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:30.831528    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ea8fcef-ee0e-41b5-905c-aa5b89044e91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-297n8\" (UID: \"5ea8fcef-ee0e-41b5-905c-aa5b89044e91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-297n8"
Nov 03 18:40:30.831800 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:30.831592    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ea8fcef-ee0e-41b5-905c-aa5b89044e91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-297n8\" (UID: \"5ea8fcef-ee0e-41b5-905c-aa5b89044e91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-297n8"
Nov 03 18:40:30.831800 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:30.831755    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jjgr\" (UniqueName: \"kubernetes.io/projected/5ea8fcef-ee0e-41b5-905c-aa5b89044e91-kube-api-access-5jjgr\") pod \"maas-api-revoke-sub-6b19338f-297n8\" (UID: \"5ea8fcef-ee0e-41b5-905c-aa5b89044e91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-297n8"
Nov 03 18:40:30.865866 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:30.865831    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:40:30.866087 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:30.865875    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgm5\""
Nov 03 18:40:30.866087 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:30.865943    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:40:30.866087 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:30.865982    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:40:30.867285 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:30.867267    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-297n8"]
Nov 03 18:40:30.933292 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:30.933259    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jjgr\" (UniqueName: \"kubernetes.io/projected/5ea8fcef-ee0e-41b5-905c-aa5b89044e91-kube-api-access-5jjgr\") pod \"maas-api-revoke-sub-6b19338f-297n8\" (UID: \"5ea8fcef-ee0e-41b5-905c-aa5b89044e91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-297n8"
Nov 03 18:40:30.933498 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:30.933355    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ea8fcef-ee0e-41b5-905c-aa5b89044e91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-297n8\" (UID: \"5ea8fcef-ee0e-41b5-905c-aa5b89044e91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-297n8"
Nov 03 18:40:30.933498 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:30.933403    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ea8fcef-ee0e-41b5-905c-aa5b89044e91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-297n8\" (UID: \"5ea8fcef-ee0e-41b5-905c-aa5b89044e91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-297n8"
Nov 03 18:40:30.934157 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:30.934132    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ea8fcef-ee0e-41b5-905c-aa5b89044e91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-297n8\" (UID: \"5ea8fcef-ee0e-41b5-905c-aa5b89044e91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-297n8"
Nov 03 18:40:30.935858 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:30.935832    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ea8fcef-ee0e-41b5-905c-aa5b89044e91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-297n8\" (UID: \"5ea8fcef-ee0e-41b5-905c-aa5b89044e91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-297n8"
Nov 03 18:40:30.994543 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:30.994512    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jjgr\" (UniqueName: \"kubernetes.io/projected/5ea8fcef-ee0e-41b5-905c-aa5b89044e91-kube-api-access-5jjgr\") pod \"maas-api-revoke-sub-6b19338f-297n8\" (UID: \"5ea8fcef-ee0e-41b5-905c-aa5b89044e91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-297n8"
Nov 03 18:40:31.074828 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:31.074785    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-297n8"
Nov 03 18:40:31.453603 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:31.453434    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-297n8"]
Nov 03 18:40:31.457389 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:40:31.457346    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ea8fcef_ee0e_41b5_905c_aa5b89044e91.slice/crio-30f4746675aedc7245b97f9cabfca4df900157cf8b39196eb1ba7068f05c67be WatchSource:0}: Error finding container 30f4746675aedc7245b97f9cabfca4df900157cf8b39196eb1ba7068f05c67be: Status 404 returned error can't find the container with id 30f4746675aedc7245b97f9cabfca4df900157cf8b39196eb1ba7068f05c67be
Nov 03 18:40:32.339660 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:32.339624    5315 generic.go:356] "Generic (PLEG): container finished" podID="5ea8fcef-ee0e-41b5-905c-aa5b89044e91" containerID="4124d874a3351d659525def1ad6061e949f712a358bc2d28b3048f6ccd35cb19" exitCode=0
Nov 03 18:40:32.340059 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:32.339703    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-297n8" event={"ID":"5ea8fcef-ee0e-41b5-905c-aa5b89044e91","Type":"ContainerDied","Data":"4124d874a3351d659525def1ad6061e949f712a358bc2d28b3048f6ccd35cb19"}
Nov 03 18:40:32.340059 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:32.339742    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-297n8" event={"ID":"5ea8fcef-ee0e-41b5-905c-aa5b89044e91","Type":"ContainerStarted","Data":"30f4746675aedc7245b97f9cabfca4df900157cf8b39196eb1ba7068f05c67be"}
Nov 03 18:40:33.470444 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:33.470416    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-297n8"
Nov 03 18:40:33.558494 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:33.558456    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ea8fcef-ee0e-41b5-905c-aa5b89044e91-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ea8fcef-ee0e-41b5-905c-aa5b89044e91-maas-api-xks-ca\") pod \"5ea8fcef-ee0e-41b5-905c-aa5b89044e91\" (UID: \"5ea8fcef-ee0e-41b5-905c-aa5b89044e91\") "
Nov 03 18:40:33.558725 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:33.558531    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ea8fcef-ee0e-41b5-905c-aa5b89044e91-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ea8fcef-ee0e-41b5-905c-aa5b89044e91-maas-api-openshift-ca\") pod \"5ea8fcef-ee0e-41b5-905c-aa5b89044e91\" (UID: \"5ea8fcef-ee0e-41b5-905c-aa5b89044e91\") "
Nov 03 18:40:33.558725 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:33.558579    5315 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ea8fcef-ee0e-41b5-905c-aa5b89044e91-kube-api-access-5jjgr\" (UniqueName: \"kubernetes.io/projected/5ea8fcef-ee0e-41b5-905c-aa5b89044e91-kube-api-access-5jjgr\") pod \"5ea8fcef-ee0e-41b5-905c-aa5b89044e91\" (UID: \"5ea8fcef-ee0e-41b5-905c-aa5b89044e91\") "
Nov 03 18:40:33.558921 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:33.558898    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ea8fcef-ee0e-41b5-905c-aa5b89044e91-maas-api-openshift-ca" pod "5ea8fcef-ee0e-41b5-905c-aa5b89044e91" (UID: "5ea8fcef-ee0e-41b5-905c-aa5b89044e91"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:40:33.560706 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:33.560682    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ea8fcef-ee0e-41b5-905c-aa5b89044e91-maas-api-xks-ca" pod "5ea8fcef-ee0e-41b5-905c-aa5b89044e91" (UID: "5ea8fcef-ee0e-41b5-905c-aa5b89044e91"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:40:33.560872 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:33.560686    5315 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ea8fcef-ee0e-41b5-905c-aa5b89044e91-kube-api-access-5jjgr" pod "5ea8fcef-ee0e-41b5-905c-aa5b89044e91" (UID: "5ea8fcef-ee0e-41b5-905c-aa5b89044e91"). InnerVolumeSpecName "kube-api-access-5jjgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:40:33.659555 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:33.659472    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ea8fcef-ee0e-41b5-905c-aa5b89044e91-maas-api-xks-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:40:33.659555 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:33.659502    5315 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ea8fcef-ee0e-41b5-905c-aa5b89044e91-maas-api-openshift-ca\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:40:33.659555 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:33.659513    5315 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jjgr\" (UniqueName: \"kubernetes.io/projected/5ea8fcef-ee0e-41b5-905c-aa5b89044e91-kube-api-access-5jjgr\") on node \"ip-10-0-128-204.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:40:34.351142 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:34.351104    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-297n8" event={"ID":"5ea8fcef-ee0e-41b5-905c-aa5b89044e91","Type":"ContainerDied","Data":"30f4746675aedc7245b97f9cabfca4df900157cf8b39196eb1ba7068f05c67be"}
Nov 03 18:40:34.351142 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:34.351134    5315 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-297n8"
Nov 03 18:40:34.351491 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:34.351155    5315 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30f4746675aedc7245b97f9cabfca4df900157cf8b39196eb1ba7068f05c67be"
Nov 03 18:40:43.072808 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:43.072773    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-c89f6d897-7c2rb_d8dacaea-58c6-4d11-ac3c-b7e234005a88/authorino/0.log"
Nov 03 18:40:53.987588 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:53.987549    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-64dd499c74-krd84_186863ea-9176-4021-a742-3d26626a88a1/manager/0.log"
Nov 03 18:40:54.578032 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:40:54.577992    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:40:56.042711 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:56.042673    5315 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-521c9b81-jwtvw"]
Nov 03 18:40:56.088538 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:56.088499    5315 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-521c9b81-jwtvw"]
Nov 03 18:40:57.102868 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:57.102833    5315 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05fd4c69-274e-4cc4-91e0-5a8a77d08705" path="/var/lib/kubelet/pods/05fd4c69-274e-4cc4-91e0-5a8a77d08705/volumes"
Nov 03 18:40:57.986064 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:57.986026    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-fjdmn_67fa6bc6-22a9-4230-a466-0a6b3929b522/revoke-keys/0.log"
Nov 03 18:40:58.197939 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:58.197896    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-297n8_5ea8fcef-ee0e-41b5-905c-aa5b89044e91/revoke-keys/0.log"
Nov 03 18:40:58.420331 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:40:58.420298    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-bn99v_64a9cfd3-025c-4ddf-9740-8516f8f091eb/revoke-keys/0.log"
Nov 03 18:41:00.175817 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:41:00.175782    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-c89f6d897-7c2rb_d8dacaea-58c6-4d11-ac3c-b7e234005a88/authorino/0.log"
Nov 03 18:41:01.743904 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:41:01.743875    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-97p65_c29d200a-c917-46e5-9cf5-21f4d952e470/ovn-acl-logging/0.log"
Nov 03 18:41:01.802335 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:41:01.802310    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-97p65_c29d200a-c917-46e5-9cf5-21f4d952e470/ovn-acl-logging/0.log"
Nov 03 18:41:03.106134 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:41:03.106093    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-bvhg2_8522e087-40fb-44bf-88e4-b8ce1ca03ec8/discovery/0.log"
Nov 03 18:41:03.683108 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:41:03.683078    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54df9ccd9f-vjs8r_1341e0b3-9782-4dae-9d05-7b77461f63ab/kube-auth-proxy/0.log"
Nov 03 18:41:03.898351 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:41:03.898312    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-4jx7c_8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8/istio-proxy/0.log"
Nov 03 18:41:04.358320 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:41:04.358286    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-7jkn9_0e37d948-2760-4441-8d44-722000f7e5b2/istio-proxy/0.log"
Nov 03 18:41:05.166858 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:41:05.166825    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-dzcvs_7e4a915b-dd46-4d77-b425-8c37440fdaad/istio-proxy/0.log"
Nov 03 18:41:05.678630 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:41:05.678582    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-nd6sq_cccc85e9-7f5c-4a60-a245-dacb32d91965/istio-proxy/0.log"
Nov 03 18:41:06.556275 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:41:06.556243    5315 scope.go:122] "RemoveContainer" containerID="28e44f806d53f48ad6a33464f2e76a86ff5ea674eaaafb5f49ec5e59e322f4b8"
Nov 03 18:41:06.658535 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:41:06.658494    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-h6mdw_fcd62b22-7b43-4f28-9f15-9066bc11e836/payload-pre-processing/0.log"
Nov 03 18:41:08.780464 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:41:08.780428    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-7nllb_4cc7a7a6-2c8d-426e-840c-3ef78e44e86a/main/0.log"
Nov 03 18:41:09.289481 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:41:09.289451    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-d6rcp_ee0c4b5d-37dd-482b-b5e9-4256a332a556/main/0.log"
Nov 03 18:41:09.561812 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:41:09.561722    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx_626f43f8-df88-48d5-9f11-ca75310a0fc2/main/0.log"
Nov 03 18:41:09.836044 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:41:09.835952    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7_69760dba-faf9-4c08-895c-ad70fe95fd49/main/0.log"
Nov 03 18:41:10.756753 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:41:10.756720    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-c89f6d897-7c2rb_d8dacaea-58c6-4d11-ac3c-b7e234005a88/authorino/0.log"
Nov 03 18:41:20.220472 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:41:20.220438    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-64dd499c74-krd84_186863ea-9176-4021-a742-3d26626a88a1/manager/0.log"
Nov 03 18:41:23.676073 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:41:23.676039    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-fjdmn_67fa6bc6-22a9-4230-a466-0a6b3929b522/revoke-keys/0.log"
Nov 03 18:41:23.890856 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:41:23.890823    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-297n8_5ea8fcef-ee0e-41b5-905c-aa5b89044e91/revoke-keys/0.log"
Nov 03 18:41:24.112633 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:41:24.112573    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-bn99v_64a9cfd3-025c-4ddf-9740-8516f8f091eb/revoke-keys/0.log"
Nov 03 18:41:24.577551 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:41:24.577514    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:41:25.480576 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:41:25.480543    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-c89f6d897-7c2rb_d8dacaea-58c6-4d11-ac3c-b7e234005a88/authorino/0.log"
Nov 03 18:41:28.110244 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:41:28.110208    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-bvhg2_8522e087-40fb-44bf-88e4-b8ce1ca03ec8/discovery/0.log"
Nov 03 18:41:28.661539 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:41:28.661503    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54df9ccd9f-vjs8r_1341e0b3-9782-4dae-9d05-7b77461f63ab/kube-auth-proxy/0.log"
Nov 03 18:41:28.891125 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:41:28.891093    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-4jx7c_8c2cfd87-4fa6-4780-8d48-4f93dd5d1dc8/istio-proxy/0.log"
Nov 03 18:41:29.360648 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:41:29.360600    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-7jkn9_0e37d948-2760-4441-8d44-722000f7e5b2/istio-proxy/0.log"
Nov 03 18:41:30.149549 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:41:30.149507    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-dzcvs_7e4a915b-dd46-4d77-b425-8c37440fdaad/istio-proxy/0.log"
Nov 03 18:41:30.634853 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:41:30.634817    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-nd6sq_cccc85e9-7f5c-4a60-a245-dacb32d91965/istio-proxy/0.log"
Nov 03 18:41:31.586700 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:41:31.586672    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-h6mdw_fcd62b22-7b43-4f28-9f15-9066bc11e836/payload-pre-processing/0.log"
Nov 03 18:41:33.500161 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:41:33.500129    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-7nllb_4cc7a7a6-2c8d-426e-840c-3ef78e44e86a/main/0.log"
Nov 03 18:41:34.032625 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:41:34.032582    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-d6rcp_ee0c4b5d-37dd-482b-b5e9-4256a332a556/main/0.log"
Nov 03 18:41:34.306927 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:41:34.306850    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-qfmpx_626f43f8-df88-48d5-9f11-ca75310a0fc2/main/0.log"
Nov 03 18:41:34.542520 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:41:34.542490    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-5b7r7_69760dba-faf9-4c08-895c-ad70fe95fd49/main/0.log"
Nov 03 18:41:54.577721 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:41:54.577630    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:42:06.576047 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:42:06.576016    5315 scope.go:122] "RemoveContainer" containerID="b2aae2ffd5e62549a98045f8b6671816f47a3202f131e03f6e3b61b245b57589"
Nov 03 18:42:06.584685 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:42:06.584664    5315 scope.go:122] "RemoveContainer" containerID="a5f29d82ccd75884a3e9db9e23f7392ab19e538acaeae871cb9aea7fddd91787"
Nov 03 18:42:24.576483 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:42:24.576442    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:42:54.577318 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:42:54.577282    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:42:55.742752 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:42:55.742716    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4xjw2_f4e4b7a9-59f2-492e-a1d1-7d97a30cdc30/global-pull-secret-syncer/0.log"
Nov 03 18:42:59.543249 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:42:59.543212    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dv4zl_b91c00dc-c14f-4b3a-b268-1eb6dcf9ad4b/konnectivity-agent/0.log"
Nov 03 18:43:00.363896 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:43:00.363861    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-204.us-west-2.compute.internal_8dbd524262c9b524921d913509ef36fa/haproxy/0.log"
Nov 03 18:43:04.771488 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:43:04.771448    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-c89f6d897-7c2rb_d8dacaea-58c6-4d11-ac3c-b7e234005a88/authorino/0.log"
Nov 03 18:43:05.581752 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:43:05.581714    5315 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-k8k5h/perf-node-gather-daemonset-kx5q9"]
Nov 03 18:43:05.582081 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:43:05.582067    5315 cpu_manager.go:414] "containerMap: removing container" podUID="5ea8fcef-ee0e-41b5-905c-aa5b89044e91" containerName="revoke-keys"
Nov 03 18:43:05.582138 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:43:05.582082    5315 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ea8fcef-ee0e-41b5-905c-aa5b89044e91" containerName="revoke-keys"
Nov 03 18:43:05.582190 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:43:05.582165    5315 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ea8fcef-ee0e-41b5-905c-aa5b89044e91" containerName="revoke-keys"
Nov 03 18:43:05.585081 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:43:05.585058    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-kx5q9"
Nov 03 18:43:05.633677 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:43:05.633639    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-k8k5h\"/\"kube-root-ca.crt\""
Nov 03 18:43:05.633677 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:43:05.633677    5315 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-k8k5h\"/\"openshift-service-ca.crt\""
Nov 03 18:43:05.680141 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:43:05.680108    5315 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-k8k5h\"/\"default-dockercfg-dhhjm\""
Nov 03 18:43:05.680361 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:43:05.680347    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-482tw\" (UniqueName: \"kubernetes.io/projected/1c91a0d5-1c2e-4fc9-b75b-c885a5a695bf-kube-api-access-482tw\") pod \"perf-node-gather-daemonset-kx5q9\" (UID: \"1c91a0d5-1c2e-4fc9-b75b-c885a5a695bf\") " pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-kx5q9"
Nov 03 18:43:05.680421 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:43:05.680382    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1c91a0d5-1c2e-4fc9-b75b-c885a5a695bf-podres\") pod \"perf-node-gather-daemonset-kx5q9\" (UID: \"1c91a0d5-1c2e-4fc9-b75b-c885a5a695bf\") " pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-kx5q9"
Nov 03 18:43:05.680501 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:43:05.680489    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c91a0d5-1c2e-4fc9-b75b-c885a5a695bf-sys\") pod \"perf-node-gather-daemonset-kx5q9\" (UID: \"1c91a0d5-1c2e-4fc9-b75b-c885a5a695bf\") " pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-kx5q9"
Nov 03 18:43:05.680542 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:43:05.680513    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1c91a0d5-1c2e-4fc9-b75b-c885a5a695bf-proc\") pod \"perf-node-gather-daemonset-kx5q9\" (UID: \"1c91a0d5-1c2e-4fc9-b75b-c885a5a695bf\") " pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-kx5q9"
Nov 03 18:43:05.680542 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:43:05.680530    5315 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1c91a0d5-1c2e-4fc9-b75b-c885a5a695bf-lib-modules\") pod \"perf-node-gather-daemonset-kx5q9\" (UID: \"1c91a0d5-1c2e-4fc9-b75b-c885a5a695bf\") " pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-kx5q9"
Nov 03 18:43:05.681393 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:43:05.681372    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k8k5h/perf-node-gather-daemonset-kx5q9"]
Nov 03 18:43:05.781394 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:43:05.781357    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c91a0d5-1c2e-4fc9-b75b-c885a5a695bf-sys\") pod \"perf-node-gather-daemonset-kx5q9\" (UID: \"1c91a0d5-1c2e-4fc9-b75b-c885a5a695bf\") " pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-kx5q9"
Nov 03 18:43:05.781394 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:43:05.781401    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1c91a0d5-1c2e-4fc9-b75b-c885a5a695bf-proc\") pod \"perf-node-gather-daemonset-kx5q9\" (UID: \"1c91a0d5-1c2e-4fc9-b75b-c885a5a695bf\") " pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-kx5q9"
Nov 03 18:43:05.781996 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:43:05.781425    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1c91a0d5-1c2e-4fc9-b75b-c885a5a695bf-lib-modules\") pod \"perf-node-gather-daemonset-kx5q9\" (UID: \"1c91a0d5-1c2e-4fc9-b75b-c885a5a695bf\") " pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-kx5q9"
Nov 03 18:43:05.781996 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:43:05.781463    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-482tw\" (UniqueName: \"kubernetes.io/projected/1c91a0d5-1c2e-4fc9-b75b-c885a5a695bf-kube-api-access-482tw\") pod \"perf-node-gather-daemonset-kx5q9\" (UID: \"1c91a0d5-1c2e-4fc9-b75b-c885a5a695bf\") " pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-kx5q9"
Nov 03 18:43:05.781996 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:43:05.781495    5315 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1c91a0d5-1c2e-4fc9-b75b-c885a5a695bf-podres\") pod \"perf-node-gather-daemonset-kx5q9\" (UID: \"1c91a0d5-1c2e-4fc9-b75b-c885a5a695bf\") " pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-kx5q9"
Nov 03 18:43:05.781996 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:43:05.781500    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c91a0d5-1c2e-4fc9-b75b-c885a5a695bf-sys\") pod \"perf-node-gather-daemonset-kx5q9\" (UID: \"1c91a0d5-1c2e-4fc9-b75b-c885a5a695bf\") " pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-kx5q9"
Nov 03 18:43:05.781996 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:43:05.781500    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1c91a0d5-1c2e-4fc9-b75b-c885a5a695bf-proc\") pod \"perf-node-gather-daemonset-kx5q9\" (UID: \"1c91a0d5-1c2e-4fc9-b75b-c885a5a695bf\") " pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-kx5q9"
Nov 03 18:43:05.781996 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:43:05.781559    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1c91a0d5-1c2e-4fc9-b75b-c885a5a695bf-lib-modules\") pod \"perf-node-gather-daemonset-kx5q9\" (UID: \"1c91a0d5-1c2e-4fc9-b75b-c885a5a695bf\") " pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-kx5q9"
Nov 03 18:43:05.781996 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:43:05.781570    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1c91a0d5-1c2e-4fc9-b75b-c885a5a695bf-podres\") pod \"perf-node-gather-daemonset-kx5q9\" (UID: \"1c91a0d5-1c2e-4fc9-b75b-c885a5a695bf\") " pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-kx5q9"
Nov 03 18:43:05.838625 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:43:05.838537    5315 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-482tw\" (UniqueName: \"kubernetes.io/projected/1c91a0d5-1c2e-4fc9-b75b-c885a5a695bf-kube-api-access-482tw\") pod \"perf-node-gather-daemonset-kx5q9\" (UID: \"1c91a0d5-1c2e-4fc9-b75b-c885a5a695bf\") " pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-kx5q9"
Nov 03 18:43:05.897293 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:43:05.897255    5315 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-kx5q9"
Nov 03 18:43:06.266914 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:43:06.266820    5315 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k8k5h/perf-node-gather-daemonset-kx5q9"]
Nov 03 18:43:06.274352 ip-10-0-128-204 kubenswrapper[5315]: W1103 18:43:06.274315    5315 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1c91a0d5_1c2e_4fc9_b75b_c885a5a695bf.slice/crio-51bc3c4a88cd975a43d7e12550505bc230251c487a7c06a4a75a0257f2eca672 WatchSource:0}: Error finding container 51bc3c4a88cd975a43d7e12550505bc230251c487a7c06a4a75a0257f2eca672: Status 404 returned error can't find the container with id 51bc3c4a88cd975a43d7e12550505bc230251c487a7c06a4a75a0257f2eca672
Nov 03 18:43:06.605707 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:43:06.605673    5315 scope.go:122] "RemoveContainer" containerID="528aacacd9cea3542e3db5e4809c2951234830dd25ccf21d3392b66020516674"
Nov 03 18:43:07.055254 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:43:07.055208    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-kx5q9" event={"ID":"1c91a0d5-1c2e-4fc9-b75b-c885a5a695bf","Type":"ContainerStarted","Data":"2af6935268fca1549562ea0d7b7beb836e087fb180624cb0982f3dafc2452a00"}
Nov 03 18:43:07.055756 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:43:07.055313    5315 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-kx5q9" event={"ID":"1c91a0d5-1c2e-4fc9-b75b-c885a5a695bf","Type":"ContainerStarted","Data":"51bc3c4a88cd975a43d7e12550505bc230251c487a7c06a4a75a0257f2eca672"}
Nov 03 18:43:07.055756 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:43:07.055346    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-kx5q9"
Nov 03 18:43:07.163745 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:43:07.163683    5315 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-kx5q9" podStartSLOduration=2.163667905 podStartE2EDuration="2.163667905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:43:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:43:07.162409658 +0000 UTC m=+3426.178607728" watchObservedRunningTime="2026-11-03 18:43:07.163667905 +0000 UTC m=+3426.179865974"
Nov 03 18:43:13.070370 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:43:13.070291    5315 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-kx5q9"
Nov 03 18:43:14.322085 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:43:14.322053    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-dwxs2_e106ef93-623a-4cb4-9848-31087f1a7711/monitoring-plugin/0.log"
Nov 03 18:43:15.946327 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:43:15.946239    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7w9wb_55b54081-644c-464e-91e1-15c2d8f5be59/node-exporter/0.log"
Nov 03 18:43:16.335339 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:43:16.335304    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7w9wb_55b54081-644c-464e-91e1-15c2d8f5be59/kube-rbac-proxy/0.log"
Nov 03 18:43:16.733732 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:43:16.733658    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7w9wb_55b54081-644c-464e-91e1-15c2d8f5be59/init-textfile/0.log"
Nov 03 18:43:22.370417 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:43:22.370382    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-4wksp_b57a39bc-800c-4ea8-a0ef-4a1721786b70/prometheus-operator/0.log"
Nov 03 18:43:22.680336 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:43:22.680242    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-4wksp_b57a39bc-800c-4ea8-a0ef-4a1721786b70/kube-rbac-proxy/0.log"
Nov 03 18:43:24.576496 ip-10-0-128-204 kubenswrapper[5315]: E1103 18:43:24.576457    5315 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qmcxl" containerName="extractor"
Nov 03 18:43:24.690562 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:43:24.690533    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74479fcb5-w59vr_d505ffde-320b-41c5-a02e-374ee52e6a1a/thanos-query/0.log"
Nov 03 18:43:25.081269 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:43:25.081233    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74479fcb5-w59vr_d505ffde-320b-41c5-a02e-374ee52e6a1a/kube-rbac-proxy-web/0.log"
Nov 03 18:43:25.471823 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:43:25.471734    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74479fcb5-w59vr_d505ffde-320b-41c5-a02e-374ee52e6a1a/kube-rbac-proxy/0.log"
Nov 03 18:43:25.862033 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:43:25.862005    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74479fcb5-w59vr_d505ffde-320b-41c5-a02e-374ee52e6a1a/prom-label-proxy/0.log"
Nov 03 18:43:26.253881 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:43:26.253808    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74479fcb5-w59vr_d505ffde-320b-41c5-a02e-374ee52e6a1a/kube-rbac-proxy-rules/0.log"
Nov 03 18:43:26.644954 ip-10-0-128-204 kubenswrapper[5315]: I1103 18:43:26.644912    5315 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74479fcb5-w59vr_d505ffde-320b-41c5-a02e-374ee52e6a1a/kube-rbac-proxy-metrics/0.log"